[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:53:12 dist-test-slave-pxl6 krb5kdc[20953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:53:12 dist-test-slave-pxl6 krb5kdc[20953](info): set up 2 sockets
Sep 28 14:53:12 dist-test-slave-pxl6 krb5kdc[20953](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 14:53:13.851348 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:53:13 dist-test-slave-pxl6 krb5kdc[20953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:53:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:53:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:53:13.981711 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:39145
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:39145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:14.066190 20966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:14.066407 20966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:14.066431 20966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:14.067685 20966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:53:14.067732 20966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:14.067747 20966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:14.067760 20966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:53:14.067773 20966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:53:14.069353 20966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:39145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:39145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:14.069738 20966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:14.070016 20966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:14.075651 20972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:14.075730 20971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:14.075855 20966 server_base.cc:1034] running on GCE node
W20260928 14:53:14.075734 20974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:14.076246 20966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:14.076545 20966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:14.077725 20966 hybrid_clock.cc:648] HybridClock initialized: now 1790607194077699 us; error 43 us; skew 500 ppm
Sep 28 14:53:14 dist-test-slave-pxl6 krb5kdc[20953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607194, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:14.080669 20966 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:53:14.081144 20966 webserver.cc:460] Webserver started at http://127.20.116.190:34279/ using document root <none> and password file <none>
I20260928 14:53:14.081414 20966 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:14.081485 20966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:14.081640 20966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:14.082880 20966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "fb5d1c1d41e248a0bcda52294a23450b"
format_stamp: "Formatted at 2026-09-28 14:53:14 on dist-test-slave-pxl6"
server_key: "bc2ed9ecc0ab0d26a2584e430db891b0"
server_key_iv: "13cf8ec026bc4a0bd409b853b1323135"
server_key_version: "encryptionkey@0"
I20260928 14:53:14.083242 20966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "fb5d1c1d41e248a0bcda52294a23450b"
format_stamp: "Formatted at 2026-09-28 14:53:14 on dist-test-slave-pxl6"
server_key: "bc2ed9ecc0ab0d26a2584e430db891b0"
server_key_iv: "13cf8ec026bc4a0bd409b853b1323135"
server_key_version: "encryptionkey@0"
I20260928 14:53:14.084719 20966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:53:14.085510 20981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:14.085819 20966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:14.085927 20966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "fb5d1c1d41e248a0bcda52294a23450b"
format_stamp: "Formatted at 2026-09-28 14:53:14 on dist-test-slave-pxl6"
server_key: "bc2ed9ecc0ab0d26a2584e430db891b0"
server_key_iv: "13cf8ec026bc4a0bd409b853b1323135"
server_key_version: "encryptionkey@0"
I20260928 14:53:14.086004 20966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:14.099084 20966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:14.101308 20966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:14.101481 20966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:14.105723 20966 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:39145
I20260928 14:53:14.105824 21033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:39145 every 8 connection(s)
I20260928 14:53:14.106202 20966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:53:14.106812 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 20966
I20260928 14:53:14.106928 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:53:14.107429 21034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:14.107729 20946 external_mini_cluster.cc:1428] Setting key 9604f3c6ea81270c887264692792bb9a
I20260928 14:53:14.110219 21034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b: Bootstrap starting.
I20260928 14:53:14.111002 21034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:14.111330 21034 log.cc:824] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:14.112085 21034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b: No bootstrap required, opened a new log
I20260928 14:53:14.114168 21034 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5d1c1d41e248a0bcda52294a23450b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39145 } }
I20260928 14:53:14.114343 21034 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:14.114374 21034 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb5d1c1d41e248a0bcda52294a23450b, State: Initialized, Role: FOLLOWER
I20260928 14:53:14.114499 21034 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5d1c1d41e248a0bcda52294a23450b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39145 } }
I20260928 14:53:14.114609 21034 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:53:14.114648 21034 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:53:14.114692 21034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:14.115303 21034 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5d1c1d41e248a0bcda52294a23450b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39145 } }
I20260928 14:53:14.115454 21034 leader_election.cc:302] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb5d1c1d41e248a0bcda52294a23450b; no voters:
I20260928 14:53:14.115653 21034 leader_election.cc:288] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:53:14.115707 21039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:14.115907 21039 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b [term 1 LEADER]: Becoming Leader. State: Replica: fb5d1c1d41e248a0bcda52294a23450b, State: Running, Role: LEADER
I20260928 14:53:14.115983 21034 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:14.116024 21039 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5d1c1d41e248a0bcda52294a23450b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39145 } }
I20260928 14:53:14.116431 21040 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb5d1c1d41e248a0bcda52294a23450b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5d1c1d41e248a0bcda52294a23450b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39145 } } }
I20260928 14:53:14.116549 21040 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:14.116801 21041 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb5d1c1d41e248a0bcda52294a23450b. Latest consensus state: current_term: 1 leader_uuid: "fb5d1c1d41e248a0bcda52294a23450b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5d1c1d41e248a0bcda52294a23450b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39145 } } }
I20260928 14:53:14.116906 21041 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b [sys.catalog]: This master's current role is: LEADER
W20260928 14:53:14.117951 21055 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:53:14.118008 21055 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:53:14.118058 21047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:53:14.118273 21047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:53:14.120012 21047 catalog_manager.cc:1347] Generated new cluster ID: 672af2ab0ec445bab36b55ecf0969646
I20260928 14:53:14.120064 21047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:53:14 dist-test-slave-pxl6 krb5kdc[20953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:14.124845 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:14.108707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53572 (local address 127.20.116.190:39145)
0928 14:53:14.110659 (+ 1952us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:14.110668 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:14.110719 (+ 51us) server_negotiation.cc:407] Connection header received
0928 14:53:14.111410 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:14.111419 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:14.111479 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:14.111625 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:14.113322 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:14.114808 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:14.116226 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:14.116876 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:14.116975 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:14.121196 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:14.121200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:14.121201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:14.121214 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:14.122985 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:14.123437 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:14.123440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:14.123440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:14.123529 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:14.123808 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:14.123809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:14.123810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:14.124046 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:14.124210 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:14.124485 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 14:53:14.124628 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1893,"thread_start_us":86,"threads_started":1}
I20260928 14:53:14.141216 21047 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:53:14.141810 21047 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:53:14.149835 21047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb5d1c1d41e248a0bcda52294a23450b: Generated new TSK 0
I20260928 14:53:14.150069 21047 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
I20260928 14:53:14.178203 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:39145
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:14.263978 21060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:14.264240 21060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:14.264277 21060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:14.265421 21060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:14.265492 21060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:14.265539 21060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:53:14.267046 21060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:39145
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:14.267331 21060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:14.267583 21060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:14.272738 21065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:14.272766 21068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:14.272738 21066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:14.273317 21060 server_base.cc:1034] running on GCE node
I20260928 14:53:14.273509 21060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:14.273754 21060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:14.274896 21060 hybrid_clock.cc:648] HybridClock initialized: now 1790607194274882 us; error 26 us; skew 500 ppm
Sep 28 14:53:14 dist-test-slave-pxl6 krb5kdc[20953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607194, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:14.277346 21060 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:53:14.277827 21060 webserver.cc:460] Webserver started at http://127.20.116.129:44813/ using document root <none> and password file <none>
I20260928 14:53:14.278017 21060 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:14.278057 21060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:14.278152 21060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:14.279285 21060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "74565c9f82d947c4b0b3581e582bc918"
format_stamp: "Formatted at 2026-09-28 14:53:14 on dist-test-slave-pxl6"
server_key: "b03125f59eef41d2d4fbd7d0c1563aac"
server_key_iv: "d1b897606a3cf6a6c824589eda6b59bb"
server_key_version: "encryptionkey@0"
I20260928 14:53:14.279610 21060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "74565c9f82d947c4b0b3581e582bc918"
format_stamp: "Formatted at 2026-09-28 14:53:14 on dist-test-slave-pxl6"
server_key: "b03125f59eef41d2d4fbd7d0c1563aac"
server_key_iv: "d1b897606a3cf6a6c824589eda6b59bb"
server_key_version: "encryptionkey@0"
I20260928 14:53:14.281052 21060 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:53:14.282132 21075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:14.282442 21060 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:14.282572 21060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "74565c9f82d947c4b0b3581e582bc918"
format_stamp: "Formatted at 2026-09-28 14:53:14 on dist-test-slave-pxl6"
server_key: "b03125f59eef41d2d4fbd7d0c1563aac"
server_key_iv: "d1b897606a3cf6a6c824589eda6b59bb"
server_key_version: "encryptionkey@0"
I20260928 14:53:14.282689 21060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:14.300540 21060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:14.303016 21060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:14.303191 21060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:14.303444 21060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:14.303870 21060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:14.303926 21060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:14.303964 21060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:14.303992 21060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:14.310143 21060 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:39087
I20260928 14:53:14.310196 21188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:39087 every 8 connection(s)
I20260928 14:53:14.310547 21060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:53:14.313891 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 21060
I20260928 14:53:14.314004 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:53:14.314160 20946 external_mini_cluster.cc:1428] Setting key 9a1b0fdfb4c56bf8fed1fdfaeb7c1086
Sep 28 14:53:14 dist-test-slave-pxl6 krb5kdc[20953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607194, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:14.321332 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:14.311290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:47385 (local address 127.20.116.190:39145)
0928 14:53:14.311356 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:14.311358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:14.311541 (+ 183us) server_negotiation.cc:407] Connection header received
0928 14:53:14.312058 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:14.312059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:14.312070 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:14.312145 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:14.313153 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:14.313653 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:14.315137 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:14.315491 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:14.315573 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:14.317942 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:14.317944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:14.317944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:14.317953 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:14.319818 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:14.320256 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:14.320258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:14.320259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:14.320329 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:14.320584 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:14.320585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:14.320585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:14.320758 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:14.320840 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:14.321105 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 14:53:14.321205 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:53:14.321376 21191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:14.311244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:39145 (local address 127.20.116.129:47385)
0928 14:53:14.311419 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:14.311463 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:14.311967 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:14.312244 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:14.312248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:14.312847 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:14.313049 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:14.313054 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:14.313790 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:14.313791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:14.315045 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:14.315048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:14.315583 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:14.315584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:14.315702 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:14.316286 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:14.316295 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:14.317851 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:14.319922 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:14.319930 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:14.319932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:14.320178 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:14.320405 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:14.320406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:14.320406 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:14.320519 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:14.320833 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:14.320838 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:14.320969 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 14:53:14.321056 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:53:14.321197 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260928 14:53:14.321960 21189 heartbeater.cc:347] Connected to a master server at 127.20.116.190:39145
I20260928 14:53:14.322086 21189 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:14.322319 21189 heartbeater.cc:511] Master 127.20.116.190:39145 requested a full tablet report, sending...
I20260928 14:53:14.323130 20998 ts_manager.cc:194] Registered new tserver with Master: 74565c9f82d947c4b0b3581e582bc918 (127.20.116.129:39087)
WARNING: no policy specified for kudu/127.20.116.130@KRBTEST.COM; defaulting to no policy
I20260928 14:53:14.323858 20998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:47385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
I20260928 14:53:14.341935 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:39145
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:14.424564 21194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:14.424798 21194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:14.424834 21194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:14.426003 21194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:14.426074 21194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:14.426126 21194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:53:14.427634 21194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:39145
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:14.427911 21194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:14.428148 21194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:14.433177 21200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:14.433229 21199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:14.433358 21194 server_base.cc:1034] running on GCE node
W20260928 14:53:14.433228 21202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:14.433612 21194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:14.433861 21194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:14.435027 21194 hybrid_clock.cc:648] HybridClock initialized: now 1790607194435012 us; error 43 us; skew 500 ppm
Sep 28 14:53:14 dist-test-slave-pxl6 krb5kdc[20953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607194, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:14.437505 21194 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:53:14.438004 21194 webserver.cc:460] Webserver started at http://127.20.116.130:37033/ using document root <none> and password file <none>
I20260928 14:53:14.438243 21194 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:14.438315 21194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:14.438468 21194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:14.439638 21194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "dc0dbcbd4f2c4812bfba7f43e81dced6"
format_stamp: "Formatted at 2026-09-28 14:53:14 on dist-test-slave-pxl6"
server_key: "e55561a2eb5b242014f4494955435c53"
server_key_iv: "2f3e4440adbe82ea3cf3e12834c935a2"
server_key_version: "encryptionkey@0"
I20260928 14:53:14.440024 21194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "dc0dbcbd4f2c4812bfba7f43e81dced6"
format_stamp: "Formatted at 2026-09-28 14:53:14 on dist-test-slave-pxl6"
server_key: "e55561a2eb5b242014f4494955435c53"
server_key_iv: "2f3e4440adbe82ea3cf3e12834c935a2"
server_key_version: "encryptionkey@0"
I20260928 14:53:14.441408 21194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:53:14.442359 21209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:14.442631 21194 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:14.442732 21194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "dc0dbcbd4f2c4812bfba7f43e81dced6"
format_stamp: "Formatted at 2026-09-28 14:53:14 on dist-test-slave-pxl6"
server_key: "e55561a2eb5b242014f4494955435c53"
server_key_iv: "2f3e4440adbe82ea3cf3e12834c935a2"
server_key_version: "encryptionkey@0"
I20260928 14:53:14.442823 21194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:14.473896 21194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:14.476140 21194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:14.476312 21194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:14.476574 21194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:14.476907 21194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:14.476961 21194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:14.477006 21194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:14.477028 21194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:14.483317 21194 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:38721
I20260928 14:53:14.483358 21322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:38721 every 8 connection(s)
I20260928 14:53:14.483697 21194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:53:14.487416 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 21194
I20260928 14:53:14.487509 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:53:14.487673 20946 external_mini_cluster.cc:1428] Setting key cf7f4b88c1710e0a3ede63637f697679
Sep 28 14:53:14 dist-test-slave-pxl6 krb5kdc[20953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607194, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:14.494624 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:14.484476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:32935 (local address 127.20.116.190:39145)
0928 14:53:14.484544 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:14.484547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:14.484760 (+ 213us) server_negotiation.cc:407] Connection header received
0928 14:53:14.485832 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:14.485833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:14.485844 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:14.485900 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:14.486511 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:14.487154 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:14.488806 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:14.489028 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:14.489107 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:14.491385 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:14.491387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:14.491387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:14.491396 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:14.493149 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:14.493560 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:14.493562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:14.493562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:14.493631 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:14.493875 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:14.493876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:14.493877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:14.494041 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:14.494120 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:14.494376 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 14:53:14.494471 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:53:14.494724 21325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:14.484462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:39145 (local address 127.20.116.130:32935)
0928 14:53:14.484631 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:14.484671 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:14.485738 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:14.485963 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:14.485966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:14.486240 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:14.486438 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:14.486442 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:14.487256 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:14.487257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:14.488709 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:14.488712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:14.489114 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:14.489115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:14.489257 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:14.489855 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:14.489864 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:14.491291 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:14.493240 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:14.493246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:14.493248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:14.493480 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:14.493706 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:14.493706 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:14.493707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:14.493803 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:14.494111 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:14.494126 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:14.494240 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 14:53:14.494325 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 14:53:14.494444 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260928 14:53:14.495363 21323 heartbeater.cc:347] Connected to a master server at 127.20.116.190:39145
I20260928 14:53:14.495481 21323 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:14.495697 21323 heartbeater.cc:511] Master 127.20.116.190:39145 requested a full tablet report, sending...
I20260928 14:53:14.496184 20998 ts_manager.cc:194] Registered new tserver with Master: dc0dbcbd4f2c4812bfba7f43e81dced6 (127.20.116.130:38721)
WARNING: no policy specified for kudu/127.20.116.131@KRBTEST.COM; defaulting to no policy
I20260928 14:53:14.496642 20998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:32935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
I20260928 14:53:14.514178 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:39145
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:14.595263 21328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:14.595523 21328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:14.595562 21328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:14.596750 21328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:14.596817 21328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:14.596863 21328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:53:14.598352 21328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:39145
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:14.598664 21328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:14.598913 21328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:14.604018 21334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:14.604018 21336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:14.604012 21333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:14.604142 21328 server_base.cc:1034] running on GCE node
I20260928 14:53:14.604355 21328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:14.604627 21328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:14.605800 21328 hybrid_clock.cc:648] HybridClock initialized: now 1790607194605788 us; error 45 us; skew 500 ppm
Sep 28 14:53:14 dist-test-slave-pxl6 krb5kdc[20953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607194, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:14.608346 21328 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:53:14.608775 21328 webserver.cc:460] Webserver started at http://127.20.116.131:38877/ using document root <none> and password file <none>
I20260928 14:53:14.609005 21328 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:14.609050 21328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:14.609145 21328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:14.610149 21328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "45dbaeb10598425f8ce4c1f9821ff03e"
format_stamp: "Formatted at 2026-09-28 14:53:14 on dist-test-slave-pxl6"
server_key: "9ebee7d727f7faa272e9e821bea0347c"
server_key_iv: "007dcd70ec384f02764ba05ffc2ccd0f"
server_key_version: "encryptionkey@0"
I20260928 14:53:14.610451 21328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "45dbaeb10598425f8ce4c1f9821ff03e"
format_stamp: "Formatted at 2026-09-28 14:53:14 on dist-test-slave-pxl6"
server_key: "9ebee7d727f7faa272e9e821bea0347c"
server_key_iv: "007dcd70ec384f02764ba05ffc2ccd0f"
server_key_version: "encryptionkey@0"
I20260928 14:53:14.611826 21328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:53:14.612644 21343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:14.612869 21328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:53:14.612944 21328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "45dbaeb10598425f8ce4c1f9821ff03e"
format_stamp: "Formatted at 2026-09-28 14:53:14 on dist-test-slave-pxl6"
server_key: "9ebee7d727f7faa272e9e821bea0347c"
server_key_iv: "007dcd70ec384f02764ba05ffc2ccd0f"
server_key_version: "encryptionkey@0"
I20260928 14:53:14.613047 21328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:14.634804 21328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:14.637154 21328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:14.637323 21328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:14.637605 21328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:14.637956 21328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:14.637995 21328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:14.638044 21328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:14.638074 21328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:14.644690 21328 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:36775
I20260928 14:53:14.644734 21456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:36775 every 8 connection(s)
I20260928 14:53:14.645130 21328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:53:14.649536 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 21328
I20260928 14:53:14.649662 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:53:14.649878 20946 external_mini_cluster.cc:1428] Setting key b494cdfd0dddd08858c3c20b948a1e56
Sep 28 14:53:14 dist-test-slave-pxl6 krb5kdc[20953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607194, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:14.655292 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:14.645974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:46635 (local address 127.20.116.190:39145)
0928 14:53:14.646057 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:14.646060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:14.646228 (+ 168us) server_negotiation.cc:407] Connection header received
0928 14:53:14.646960 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:14.646962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:14.646974 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:14.647035 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:14.648119 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:14.648687 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:14.649902 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:14.650217 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:14.650324 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:14.652729 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:14.652731 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:14.652731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:14.652739 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:14.654015 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:14.654406 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:14.654406 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:14.654407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:14.654457 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:14.654732 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:14.654733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:14.654733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:14.654843 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:14.654898 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:14.655159 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 14:53:14.655224 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:53:14.655436 21459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:14.645909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:39145 (local address 127.20.116.131:46635)
0928 14:53:14.646102 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:14.646142 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:14.646826 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:14.647128 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:14.647131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:14.647770 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:14.648009 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:14.648014 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:14.648796 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:14.648797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:14.649742 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:14.649746 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:14.650458 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:14.650459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:14.650650 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:14.651213 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:14.651221 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:14.652640 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:14.654094 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:14.654100 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:14.654102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:14.654338 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:14.654517 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:14.654518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:14.654518 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:14.654673 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:14.654899 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:14.654903 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:14.655024 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 14:53:14.655103 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 14:53:14.655229 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":97,"threads_started":1}
I20260928 14:53:14.655952 21457 heartbeater.cc:347] Connected to a master server at 127.20.116.190:39145
I20260928 14:53:14.656065 21457 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:14.656257 21457 heartbeater.cc:511] Master 127.20.116.190:39145 requested a full tablet report, sending...
I20260928 14:53:14.656654 20998 ts_manager.cc:194] Registered new tserver with Master: 45dbaeb10598425f8ce4c1f9821ff03e (127.20.116.131:36775)
I20260928 14:53:14.657030 20998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:46635
I20260928 14:53:14.661882 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:53:14 dist-test-slave-pxl6 krb5kdc[20953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607194, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:53:14 dist-test-slave-pxl6 krb5kdc[20953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607194, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.116.129@KRBTEST.COM
I20260928 14:53:14.685482 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:14.677490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60846 (local address 127.20.116.129:39087)
0928 14:53:14.677693 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:14.677699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:14.677710 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:53:14.677788 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:14.677791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:14.677805 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:14.677874 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:14.678400 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:14.679199 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:14.679906 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:14.680242 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:14.680297 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:14.682748 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:14.682758 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:14.682758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:14.682765 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:14.684298 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:14.684666 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:14.684669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:14.684670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:14.684729 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:14.684903 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:14.684904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:14.684904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:14.685081 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:14.685219 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:14.685274 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 14:53:14.685343 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":120,"threads_started":1}
W20260928 14:53:14.685904 21103 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60846
Sep 28 14:53:14 dist-test-slave-pxl6 krb5kdc[20953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:53:14 dist-test-slave-pxl6 krb5kdc[20953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.129@KRBTEST.COM
I20260928 14:53:14.709331 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:14.701775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60856 (local address 127.20.116.129:39087)
0928 14:53:14.701853 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:14.701855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:14.701944 (+ 89us) server_negotiation.cc:407] Connection header received
0928 14:53:14.701994 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:14.701995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:14.702008 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:14.702060 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:14.702763 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:14.703286 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:14.703989 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:14.704196 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:14.704252 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:14.706657 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:14.706659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:14.706659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:14.706666 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:14.708242 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:14.708584 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:14.708585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:14.708586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:14.708640 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:14.708883 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:14.708884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:14.708884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:14.708978 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:14.709071 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:14.709173 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 14:53:14.709236 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:53:14.710052 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 21060
I20260928 14:53:14.716717 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 21194
I20260928 14:53:14.723206 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 21328
I20260928 14:53:14.727999 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 20966
2026-09-28T14:53:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1940 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:53:14 dist-test-slave-pxl6 krb5kdc[21471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:53:14 dist-test-slave-pxl6 krb5kdc[21471](info): set up 2 sockets
Sep 28 14:53:14 dist-test-slave-pxl6 krb5kdc[21471](info): commencing operation
krb5kdc: starting...
W20260928 14:53:15.771001 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:53:15 dist-test-slave-pxl6 krb5kdc[21471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:53:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:53:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:53:15.894783 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:38445
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:38313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:38445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:15.979086 21484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:15.979337 21484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:15.979373 21484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:15.980561 21484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:53:15.980638 21484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:15.980662 21484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:15.980692 21484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:53:15.980723 21484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:53:15.982208 21484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:38313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:38445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:38445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:15.982492 21484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:15.982754 21484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:15.987900 21489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:15.988037 21484 server_base.cc:1034] running on GCE node
W20260928 14:53:15.987905 21490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:15.987995 21492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:15.988407 21484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:15.988734 21484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:15.989893 21484 hybrid_clock.cc:648] HybridClock initialized: now 1790607195989876 us; error 32 us; skew 500 ppm
Sep 28 14:53:15 dist-test-slave-pxl6 krb5kdc[21471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607195, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:15.992442 21484 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:53:15.992944 21484 webserver.cc:460] Webserver started at http://127.20.116.190:32941/ using document root <none> and password file <none>
I20260928 14:53:15.993182 21484 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:15.993251 21484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:15.993407 21484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:15.994513 21484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "6ba71821dde94e769efe9289f527629d"
format_stamp: "Formatted at 2026-09-28 14:53:15 on dist-test-slave-pxl6"
server_key: "9cf1b5e044f1c70ae9a14764fd09e035"
server_key_iv: "51b2180bf0e9b332f411b1622cc22a2f"
server_key_version: "encryptionkey@0"
I20260928 14:53:15.994963 21484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "6ba71821dde94e769efe9289f527629d"
format_stamp: "Formatted at 2026-09-28 14:53:15 on dist-test-slave-pxl6"
server_key: "9cf1b5e044f1c70ae9a14764fd09e035"
server_key_iv: "51b2180bf0e9b332f411b1622cc22a2f"
server_key_version: "encryptionkey@0"
I20260928 14:53:15.996359 21484 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:53:15.997152 21499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:15.997359 21484 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:53:15.997473 21484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "6ba71821dde94e769efe9289f527629d"
format_stamp: "Formatted at 2026-09-28 14:53:15 on dist-test-slave-pxl6"
server_key: "9cf1b5e044f1c70ae9a14764fd09e035"
server_key_iv: "51b2180bf0e9b332f411b1622cc22a2f"
server_key_version: "encryptionkey@0"
I20260928 14:53:15.997551 21484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:16.026347 21484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:16.028616 21484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:16.028803 21484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:16.033164 21484 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:38445
I20260928 14:53:16.033192 21551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:38445 every 8 connection(s)
I20260928 14:53:16.033577 21484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:53:16.034474 21552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:16.037246 21552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d: Bootstrap starting.
I20260928 14:53:16.037969 21552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:16.038378 21552 log.cc:824] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:16.039289 21552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d: No bootstrap required, opened a new log
I20260928 14:53:16.040314 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 21484
I20260928 14:53:16.040383 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:53:16.040557 20946 external_mini_cluster.cc:1428] Setting key b6db9fca6edbed20c38b6d4ed723ca1f
I20260928 14:53:16.041278 21552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba71821dde94e769efe9289f527629d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 38445 } }
I20260928 14:53:16.041487 21552 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:16.041567 21552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ba71821dde94e769efe9289f527629d, State: Initialized, Role: FOLLOWER
I20260928 14:53:16.041713 21552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba71821dde94e769efe9289f527629d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 38445 } }
I20260928 14:53:16.041798 21552 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:53:16.041838 21552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:53:16.041882 21552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:16.042488 21552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba71821dde94e769efe9289f527629d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 38445 } }
I20260928 14:53:16.042666 21552 leader_election.cc:302] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ba71821dde94e769efe9289f527629d; no voters:
I20260928 14:53:16.042848 21552 leader_election.cc:288] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:53:16.042881 21557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:16.043015 21557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d [term 1 LEADER]: Becoming Leader. State: Replica: 6ba71821dde94e769efe9289f527629d, State: Running, Role: LEADER
I20260928 14:53:16.043141 21552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:16.043112 21557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba71821dde94e769efe9289f527629d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 38445 } }
I20260928 14:53:16.043520 21558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ba71821dde94e769efe9289f527629d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba71821dde94e769efe9289f527629d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 38445 } } }
I20260928 14:53:16.043629 21558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:16.043509 21557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ba71821dde94e769efe9289f527629d. Latest consensus state: current_term: 1 leader_uuid: "6ba71821dde94e769efe9289f527629d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba71821dde94e769efe9289f527629d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 38445 } } }
I20260928 14:53:16.043790 21557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:16.044226 21563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:53:16.044777 21563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:53:16.046945 21563 catalog_manager.cc:1347] Generated new cluster ID: 3a897ca8095b414c8d5fab6eea8f7208
I20260928 14:53:16.047009 21563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:53:16 dist-test-slave-pxl6 krb5kdc[21471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:16.052235 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:16.041193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42776 (local address 127.20.116.190:38445)
0928 14:53:16.041415 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:16.041424 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:16.041456 (+ 32us) server_negotiation.cc:407] Connection header received
0928 14:53:16.041588 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:16.041597 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:16.041660 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:16.041803 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:16.042585 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.044008 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.045080 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.045710 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.045824 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:16.048493 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:16.048497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:16.048498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:16.048512 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:16.050566 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:16.050935 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:16.050938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:16.050939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:16.051058 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:16.051288 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:16.051289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:16.051289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:16.051579 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:16.051765 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:16.051843 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 14:53:16.051998 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":98,"threads_started":1}
I20260928 14:53:16.059345 21563 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:53:16.060096 21563 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:53:16.064327 21563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ba71821dde94e769efe9289f527629d: Generated new TSK 0
I20260928 14:53:16.064531 21563 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
I20260928 14:53:16.098994 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:38445
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:38313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:16.182282 21578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:16.182507 21578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:16.182559 21578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:16.183768 21578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:16.183815 21578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:16.183841 21578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:53:16.185426 21578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:38313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:38445
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:16.185653 21578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:16.185837 21578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:16.191012 21583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:16.191005 21586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:16.191053 21584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:16.191211 21578 server_base.cc:1034] running on GCE node
I20260928 14:53:16.191443 21578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:16.191665 21578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:16.192837 21578 hybrid_clock.cc:648] HybridClock initialized: now 1790607196192810 us; error 43 us; skew 500 ppm
Sep 28 14:53:16 dist-test-slave-pxl6 krb5kdc[21471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607196, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:16.195616 21578 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:53:16.196074 21578 webserver.cc:460] Webserver started at http://127.20.116.129:42705/ using document root <none> and password file <none>
I20260928 14:53:16.196264 21578 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:16.196305 21578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:16.196398 21578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:16.197443 21578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "082b64b488b8440badb1ddb7ad263b74"
format_stamp: "Formatted at 2026-09-28 14:53:16 on dist-test-slave-pxl6"
server_key: "e6afc51c5b176a04375a0e64e95944c2"
server_key_iv: "d4a834d94969f9769bb68c61b41a335c"
server_key_version: "encryptionkey@0"
I20260928 14:53:16.197757 21578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "082b64b488b8440badb1ddb7ad263b74"
format_stamp: "Formatted at 2026-09-28 14:53:16 on dist-test-slave-pxl6"
server_key: "e6afc51c5b176a04375a0e64e95944c2"
server_key_iv: "d4a834d94969f9769bb68c61b41a335c"
server_key_version: "encryptionkey@0"
I20260928 14:53:16.199136 21578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:53:16.199819 21593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:16.200007 21578 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:53:16.200090 21578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "082b64b488b8440badb1ddb7ad263b74"
format_stamp: "Formatted at 2026-09-28 14:53:16 on dist-test-slave-pxl6"
server_key: "e6afc51c5b176a04375a0e64e95944c2"
server_key_iv: "d4a834d94969f9769bb68c61b41a335c"
server_key_version: "encryptionkey@0"
I20260928 14:53:16.200145 21578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:16.208683 21578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:16.211007 21578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:16.211122 21578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:16.211300 21578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:16.211620 21578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:16.211655 21578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:16.211678 21578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:16.211690 21578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:16.217633 21578 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:42581
I20260928 14:53:16.217710 21706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:42581 every 8 connection(s)
I20260928 14:53:16.217998 21578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:53:16 dist-test-slave-pxl6 krb5kdc[21471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607196, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:16.224253 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 21578
I20260928 14:53:16.224331 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:53:16.224486 20946 external_mini_cluster.cc:1428] Setting key cc85ef36713d402e1d70244ec3736ee8
I20260928 14:53:16.227726 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:16.218722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:38445 (local address 127.20.116.129:43247)
0928 14:53:16.218901 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:16.218936 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:16.219819 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:16.220044 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:16.220047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:16.220318 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:16.220503 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:16.220508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.221167 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.221168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:16.221989 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:16.221991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.222392 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.222393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:16.222497 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:16.222999 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:16.223007 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:16.224441 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:16.226260 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:16.226268 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:16.226270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:16.226504 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:16.226785 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:16.226786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:16.226786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:16.226898 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:16.227209 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:16.227214 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:16.227333 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 14:53:16.227418 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 14:53:16.227548 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":98,"threads_started":1}
I20260928 14:53:16.227732 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:16.218754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:43247 (local address 127.20.116.190:38445)
0928 14:53:16.218828 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:16.218831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:16.219015 (+ 184us) server_negotiation.cc:407] Connection header received
0928 14:53:16.219919 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:16.219920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:16.219935 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:16.219974 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:16.220576 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.221085 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.222088 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.222307 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.222381 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:16.224580 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:16.224582 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:16.224582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:16.224591 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:16.226174 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:16.226634 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:16.226635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:16.226636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:16.226709 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:16.226972 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:16.226973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:16.226973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:16.227139 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:16.227214 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:16.227466 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 14:53:16.227610 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:53:16.228322 21707 heartbeater.cc:347] Connected to a master server at 127.20.116.190:38445
I20260928 14:53:16.228440 21707 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:16.228638 21707 heartbeater.cc:511] Master 127.20.116.190:38445 requested a full tablet report, sending...
I20260928 14:53:16.229322 21516 ts_manager.cc:194] Registered new tserver with Master: 082b64b488b8440badb1ddb7ad263b74 (127.20.116.129:42581)
I20260928 14:53:16.230345 21516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:43247
WARNING: no policy specified for kudu/127.20.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
I20260928 14:53:16.253424 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:38445
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:38313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:16.351495 21712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:16.351747 21712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:16.351786 21712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:16.353005 21712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:16.353072 21712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:16.353119 21712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:53:16.354736 21712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:38313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:38445
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:16.355017 21712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:16.355253 21712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:16.360743 21721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:16.360754 21719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:16.360955 21712 server_base.cc:1034] running on GCE node
W20260928 14:53:16.360786 21718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:16.361238 21712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:16.361461 21712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:16.362640 21712 hybrid_clock.cc:648] HybridClock initialized: now 1790607196362608 us; error 43 us; skew 500 ppm
Sep 28 14:53:16 dist-test-slave-pxl6 krb5kdc[21471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607196, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:16.365477 21712 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:53:16.366001 21712 webserver.cc:460] Webserver started at http://127.20.116.130:34791/ using document root <none> and password file <none>
I20260928 14:53:16.366283 21712 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:16.366343 21712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:16.366501 21712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:16.367650 21712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "0dabbab44cec4d378eb643db561fc29a"
format_stamp: "Formatted at 2026-09-28 14:53:16 on dist-test-slave-pxl6"
server_key: "835b3c8f1369cc18614beed6eeabbaac"
server_key_iv: "7ef0902d9e599133b6e10f3093775f9d"
server_key_version: "encryptionkey@0"
I20260928 14:53:16.368026 21712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "0dabbab44cec4d378eb643db561fc29a"
format_stamp: "Formatted at 2026-09-28 14:53:16 on dist-test-slave-pxl6"
server_key: "835b3c8f1369cc18614beed6eeabbaac"
server_key_iv: "7ef0902d9e599133b6e10f3093775f9d"
server_key_version: "encryptionkey@0"
I20260928 14:53:16.369431 21712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:53:16.370312 21728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:16.370563 21712 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:16.370661 21712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "0dabbab44cec4d378eb643db561fc29a"
format_stamp: "Formatted at 2026-09-28 14:53:16 on dist-test-slave-pxl6"
server_key: "835b3c8f1369cc18614beed6eeabbaac"
server_key_iv: "7ef0902d9e599133b6e10f3093775f9d"
server_key_version: "encryptionkey@0"
I20260928 14:53:16.370718 21712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:16.383391 21712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:16.385391 21712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:16.385505 21712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:16.385782 21712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:16.386096 21712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:16.386129 21712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:16.386157 21712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:16.386175 21712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:16.392120 21712 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:46595
I20260928 14:53:16.392210 21841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:46595 every 8 connection(s)
I20260928 14:53:16.392527 21712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:53:16.398752 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 21712
I20260928 14:53:16.398831 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
Sep 28 14:53:16 dist-test-slave-pxl6 krb5kdc[21471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607196, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:16.399016 20946 external_mini_cluster.cc:1428] Setting key a97116a53943e6324b61c4fcc4819086
I20260928 14:53:16.402803 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:16.393335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:55467 (local address 127.20.116.190:38445)
0928 14:53:16.393383 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:16.393385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:16.393646 (+ 261us) server_negotiation.cc:407] Connection header received
0928 14:53:16.394662 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:16.394664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:16.394685 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:16.394740 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:16.395361 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.395870 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.396945 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.397148 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.397203 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:16.399586 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:16.399587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:16.399588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:16.399596 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:16.401200 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:16.401668 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:16.401670 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:16.401670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:16.401737 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:16.402028 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:16.402029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:16.402029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:16.402163 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:16.402243 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:16.402581 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 14:53:16.402707 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:53:16.403143 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:16.393308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:38445 (local address 127.20.116.130:55467)
0928 14:53:16.393490 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:16.393540 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:16.394485 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:16.394803 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:16.394806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:16.395083 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:16.395283 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:16.395287 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.395951 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.395952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:16.396863 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:16.396865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.397229 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.397230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:16.397346 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:16.397842 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:16.397851 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:16.399477 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:16.401292 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:16.401302 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:16.401305 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:16.401561 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:16.401814 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:16.401815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:16.401815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:16.401951 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:16.402233 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:16.402242 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:16.402407 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 14:53:16.402522 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 14:53:16.402741 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260928 14:53:16.403702 21842 heartbeater.cc:347] Connected to a master server at 127.20.116.190:38445
I20260928 14:53:16.403805 21842 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:16.404011 21842 heartbeater.cc:511] Master 127.20.116.190:38445 requested a full tablet report, sending...
I20260928 14:53:16.404459 21516 ts_manager.cc:194] Registered new tserver with Master: 0dabbab44cec4d378eb643db561fc29a (127.20.116.130:46595)
I20260928 14:53:16.404857 21516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:55467
WARNING: no policy specified for kudu/127.20.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
I20260928 14:53:16.426604 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:38445
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:38313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:16.511799 21847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:16.512007 21847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:16.512032 21847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:16.513123 21847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:16.513157 21847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:16.513183 21847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:53:16.514607 21847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:38313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:38445
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:16.514801 21847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:16.514976 21847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:53:16.520129 21847 server_base.cc:1034] running on GCE node
W20260928 14:53:16.520109 21853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:16.520112 21852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:16.520112 21855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:16.520525 21847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:16.520746 21847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:16.521891 21847 hybrid_clock.cc:648] HybridClock initialized: now 1790607196521857 us; error 48 us; skew 500 ppm
Sep 28 14:53:16 dist-test-slave-pxl6 krb5kdc[21471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607196, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:16.524413 21847 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:53:16.524801 21847 webserver.cc:460] Webserver started at http://127.20.116.131:42607/ using document root <none> and password file <none>
I20260928 14:53:16.524986 21847 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:16.525025 21847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:16.525112 21847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:16.526172 21847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "cfc94c7d3138458bb152b634388a6665"
format_stamp: "Formatted at 2026-09-28 14:53:16 on dist-test-slave-pxl6"
server_key: "e6a0fcc37a0a0cab9045ac39e7eebb5c"
server_key_iv: "64a9f1cb2339f1300875d5391b0f9a9f"
server_key_version: "encryptionkey@0"
I20260928 14:53:16.526469 21847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "cfc94c7d3138458bb152b634388a6665"
format_stamp: "Formatted at 2026-09-28 14:53:16 on dist-test-slave-pxl6"
server_key: "e6a0fcc37a0a0cab9045ac39e7eebb5c"
server_key_iv: "64a9f1cb2339f1300875d5391b0f9a9f"
server_key_version: "encryptionkey@0"
I20260928 14:53:16.527876 21847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:53:16.528644 21862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:16.528873 21847 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:53:16.528983 21847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "cfc94c7d3138458bb152b634388a6665"
format_stamp: "Formatted at 2026-09-28 14:53:16 on dist-test-slave-pxl6"
server_key: "e6a0fcc37a0a0cab9045ac39e7eebb5c"
server_key_iv: "64a9f1cb2339f1300875d5391b0f9a9f"
server_key_version: "encryptionkey@0"
I20260928 14:53:16.529059 21847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:16.548547 21847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:16.550616 21847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:16.550778 21847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:16.551015 21847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:16.551331 21847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:16.551385 21847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:16.551427 21847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:16.551455 21847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:16.557646 21847 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:34615
I20260928 14:53:16.557724 21975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:34615 every 8 connection(s)
I20260928 14:53:16.558079 21847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:53:16.561748 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 21847
I20260928 14:53:16.561848 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:53:16.561996 20946 external_mini_cluster.cc:1428] Setting key cc8ad6e950202681ba6f8613cdc49176
Sep 28 14:53:16 dist-test-slave-pxl6 krb5kdc[21471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607196, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:16.568378 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:16.558961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:45333 (local address 127.20.116.190:38445)
0928 14:53:16.559050 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:16.559052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:16.559226 (+ 174us) server_negotiation.cc:407] Connection header received
0928 14:53:16.560182 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:16.560183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:16.560195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:16.560248 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:16.560845 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.561336 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.562887 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.563124 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.563185 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:16.565574 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:16.565576 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:16.565576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:16.565584 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:16.566924 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:16.567387 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:16.567389 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:16.567390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:16.567455 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:16.567712 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:16.567713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:16.567713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:16.567834 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:16.567912 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:16.568199 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 14:53:16.568283 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:53:16.568601 21978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:16.558919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:38445 (local address 127.20.116.131:45333)
0928 14:53:16.559091 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:16.559130 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:16.560072 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:16.560304 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:16.560307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:16.560569 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:16.560776 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:16.560780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.561415 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.561416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:16.562803 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:16.562807 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.563201 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.563202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:16.563307 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:16.563876 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:16.563884 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:16.565457 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:16.567022 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:16.567028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:16.567031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:16.567288 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:16.567525 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:16.567526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:16.567527 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:16.567642 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:16.567902 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:16.567908 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:16.568053 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 14:53:16.568164 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 14:53:16.568330 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
I20260928 14:53:16.569222 21976 heartbeater.cc:347] Connected to a master server at 127.20.116.190:38445
I20260928 14:53:16.569327 21976 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:16.569499 21976 heartbeater.cc:511] Master 127.20.116.190:38445 requested a full tablet report, sending...
I20260928 14:53:16.569936 21516 ts_manager.cc:194] Registered new tserver with Master: cfc94c7d3138458bb152b634388a6665 (127.20.116.131:34615)
I20260928 14:53:16.570349 21516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:45333
I20260928 14:53:16.573781 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:53:16.581723 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:16.575411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42788 (local address 127.20.116.190:38445)
0928 14:53:16.575493 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:16.575495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:16.575546 (+ 51us) server_negotiation.cc:407] Connection header received
0928 14:53:16.575623 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:16.575624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:16.575633 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:16.575682 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:16.576190 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.576713 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.577462 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.577704 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.577769 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:16.579044 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:16.579045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:16.579046 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:16.579052 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:16.580465 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:16.581005 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:16.581007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:16.581007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:16.581069 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:16.581255 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:16.581256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:16.581256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:16.581402 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:16.581477 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:16.581588 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 14:53:16.581655 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:53:16.584244 21516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:53:16.586050 21516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:53:16.593518 21992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:16.589183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57088 (local address 127.20.116.130:46595)
0928 14:53:16.589661 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:16.589664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:16.589671 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:53:16.589701 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:16.589702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:16.589713 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:16.589798 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:16.589996 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.590905 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.592546 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.593074 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.593171 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:16.593229 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:16.593398 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 14:53:16.593442 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":68,"threads_started":1}
I20260928 14:53:16.593509 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:16.588798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:46595 (local address 127.0.0.1:57088)
0928 14:53:16.589169 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:16.589180 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:16.589281 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:16.589812 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:16.589814 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:16.589816 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:16.589898 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:16.589900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.590999 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.591000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:16.592446 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:16.592449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.593165 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.593166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:16.593256 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:16.593270 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 14:53:16.593364 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 14:53:16.593418 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":39,"thread_start_us":94,"threads_started":1}
I20260928 14:53:16.593511 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:16.589048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:34615 (local address 127.0.0.1:42734)
0928 14:53:16.589907 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:16.589917 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:16.589967 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:16.590263 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:16.590264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:16.590266 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:16.590310 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:16.590312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.591564 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.591565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:16.592510 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:16.592513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.593186 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.593187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:16.593260 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:16.593284 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:53:16.593344 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 14:53:16.593400 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260928 14:53:16.594204 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:16.589141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42734 (local address 127.20.116.131:34615)
0928 14:53:16.589352 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:16.589357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:16.590028 (+ 671us) server_negotiation.cc:407] Connection header received
0928 14:53:16.590086 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:16.590088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:16.590103 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:16.590174 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:16.590394 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.591406 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.592592 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.593073 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.593165 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:16.593229 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:16.594069 (+ 840us) server_negotiation.cc:299] Negotiation successful
0928 14:53:16.594126 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":97,"threads_started":1}
I20260928 14:53:16.594363 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:16.588781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:42581 (local address 127.0.0.1:36196)
0928 14:53:16.588907 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:16.588953 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:16.589022 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:16.589594 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:16.589597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:16.589600 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:16.589675 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:16.589677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.591130 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.591130 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:16.592446 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:16.592449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.594123 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.594124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:16.594186 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:16.594191 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:53:16.594233 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 14:53:16.594281 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":63,"thread_start_us":73,"threads_started":1}
I20260928 14:53:16.594463 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:16.588934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36196 (local address 127.20.116.129:42581)
0928 14:53:16.589392 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:16.589396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:16.589406 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:53:16.589443 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:16.589445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:16.589459 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:16.589531 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:16.589791 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.591020 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.593271 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.594018 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.594110 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:16.594297 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:16.594327 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 14:53:16.594370 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":99,"threads_started":1}
I20260928 14:53:16.594748 21776 tablet_service.cc:1467] Processing CreateTablet for tablet b9ac96fd73a94cecbf837520e3206564 (DEFAULT_TABLE table=test-table [id=4679ff447185474b81d4f92041e40bd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:16.595158 21776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9ac96fd73a94cecbf837520e3206564. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:16.595252 21910 tablet_service.cc:1467] Processing CreateTablet for tablet b9ac96fd73a94cecbf837520e3206564 (DEFAULT_TABLE table=test-table [id=4679ff447185474b81d4f92041e40bd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:16.595669 21910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9ac96fd73a94cecbf837520e3206564. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:16.597908 21993 tablet_bootstrap.cc:492] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a: Bootstrap starting.
I20260928 14:53:16.598242 21994 tablet_bootstrap.cc:492] T b9ac96fd73a94cecbf837520e3206564 P cfc94c7d3138458bb152b634388a6665: Bootstrap starting.
I20260928 14:53:16.598593 21641 tablet_service.cc:1467] Processing CreateTablet for tablet b9ac96fd73a94cecbf837520e3206564 (DEFAULT_TABLE table=test-table [id=4679ff447185474b81d4f92041e40bd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:16.598742 21993 tablet_bootstrap.cc:654] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:16.598950 21641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9ac96fd73a94cecbf837520e3206564. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:16.598974 21994 tablet_bootstrap.cc:654] T b9ac96fd73a94cecbf837520e3206564 P cfc94c7d3138458bb152b634388a6665: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:16.599432 21993 log.cc:824] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:16.599470 21994 log.cc:824] T b9ac96fd73a94cecbf837520e3206564 P cfc94c7d3138458bb152b634388a6665: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:16.600253 21994 tablet_bootstrap.cc:492] T b9ac96fd73a94cecbf837520e3206564 P cfc94c7d3138458bb152b634388a6665: No bootstrap required, opened a new log
I20260928 14:53:16.600342 21994 ts_tablet_manager.cc:1397] T b9ac96fd73a94cecbf837520e3206564 P cfc94c7d3138458bb152b634388a6665: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:53:16.602020 21997 tablet_bootstrap.cc:492] T b9ac96fd73a94cecbf837520e3206564 P 082b64b488b8440badb1ddb7ad263b74: Bootstrap starting.
I20260928 14:53:16.602206 21994 raft_consensus.cc:348] T b9ac96fd73a94cecbf837520e3206564 P cfc94c7d3138458bb152b634388a6665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082b64b488b8440badb1ddb7ad263b74" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 42581 } } peers { permanent_uuid: "0dabbab44cec4d378eb643db561fc29a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 46595 } } peers { permanent_uuid: "cfc94c7d3138458bb152b634388a6665" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34615 } }
I20260928 14:53:16.602411 21994 raft_consensus.cc:374] T b9ac96fd73a94cecbf837520e3206564 P cfc94c7d3138458bb152b634388a6665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:16.602447 21994 raft_consensus.cc:729] T b9ac96fd73a94cecbf837520e3206564 P cfc94c7d3138458bb152b634388a6665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc94c7d3138458bb152b634388a6665, State: Initialized, Role: FOLLOWER
I20260928 14:53:16.602697 21993 tablet_bootstrap.cc:492] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a: No bootstrap required, opened a new log
I20260928 14:53:16.602645 21994 consensus_queue.cc:260] T b9ac96fd73a94cecbf837520e3206564 P cfc94c7d3138458bb152b634388a6665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082b64b488b8440badb1ddb7ad263b74" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 42581 } } peers { permanent_uuid: "0dabbab44cec4d378eb643db561fc29a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 46595 } } peers { permanent_uuid: "cfc94c7d3138458bb152b634388a6665" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34615 } }
I20260928 14:53:16.602759 21997 tablet_bootstrap.cc:654] T b9ac96fd73a94cecbf837520e3206564 P 082b64b488b8440badb1ddb7ad263b74: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:16.602792 21993 ts_tablet_manager.cc:1397] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 14:53:16.602979 21994 ts_tablet_manager.cc:1428] T b9ac96fd73a94cecbf837520e3206564 P cfc94c7d3138458bb152b634388a6665: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:53:16.603138 21976 heartbeater.cc:503] Master 127.20.116.190:38445 was elected leader, sending a full tablet report...
I20260928 14:53:16.603407 21997 log.cc:824] T b9ac96fd73a94cecbf837520e3206564 P 082b64b488b8440badb1ddb7ad263b74: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:16.604485 21997 tablet_bootstrap.cc:492] T b9ac96fd73a94cecbf837520e3206564 P 082b64b488b8440badb1ddb7ad263b74: No bootstrap required, opened a new log
I20260928 14:53:16.604563 21997 ts_tablet_manager.cc:1397] T b9ac96fd73a94cecbf837520e3206564 P 082b64b488b8440badb1ddb7ad263b74: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:53:16.604875 21993 raft_consensus.cc:348] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082b64b488b8440badb1ddb7ad263b74" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 42581 } } peers { permanent_uuid: "0dabbab44cec4d378eb643db561fc29a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 46595 } } peers { permanent_uuid: "cfc94c7d3138458bb152b634388a6665" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34615 } }
I20260928 14:53:16.605069 21993 raft_consensus.cc:374] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:16.605105 21993 raft_consensus.cc:729] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dabbab44cec4d378eb643db561fc29a, State: Initialized, Role: FOLLOWER
I20260928 14:53:16.605224 21993 consensus_queue.cc:260] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082b64b488b8440badb1ddb7ad263b74" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 42581 } } peers { permanent_uuid: "0dabbab44cec4d378eb643db561fc29a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 46595 } } peers { permanent_uuid: "cfc94c7d3138458bb152b634388a6665" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34615 } }
I20260928 14:53:16.605499 21993 ts_tablet_manager.cc:1428] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:53:16.605664 21842 heartbeater.cc:503] Master 127.20.116.190:38445 was elected leader, sending a full tablet report...
I20260928 14:53:16.606525 21997 raft_consensus.cc:348] T b9ac96fd73a94cecbf837520e3206564 P 082b64b488b8440badb1ddb7ad263b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082b64b488b8440badb1ddb7ad263b74" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 42581 } } peers { permanent_uuid: "0dabbab44cec4d378eb643db561fc29a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 46595 } } peers { permanent_uuid: "cfc94c7d3138458bb152b634388a6665" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34615 } }
I20260928 14:53:16.606755 21997 raft_consensus.cc:374] T b9ac96fd73a94cecbf837520e3206564 P 082b64b488b8440badb1ddb7ad263b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:16.606796 21997 raft_consensus.cc:729] T b9ac96fd73a94cecbf837520e3206564 P 082b64b488b8440badb1ddb7ad263b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 082b64b488b8440badb1ddb7ad263b74, State: Initialized, Role: FOLLOWER
I20260928 14:53:16.606917 21997 consensus_queue.cc:260] T b9ac96fd73a94cecbf837520e3206564 P 082b64b488b8440badb1ddb7ad263b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082b64b488b8440badb1ddb7ad263b74" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 42581 } } peers { permanent_uuid: "0dabbab44cec4d378eb643db561fc29a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 46595 } } peers { permanent_uuid: "cfc94c7d3138458bb152b634388a6665" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34615 } }
I20260928 14:53:16.607225 21997 ts_tablet_manager.cc:1428] T b9ac96fd73a94cecbf837520e3206564 P 082b64b488b8440badb1ddb7ad263b74: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:53:16.607239 21707 heartbeater.cc:503] Master 127.20.116.190:38445 was elected leader, sending a full tablet report...
I20260928 14:53:16.611595 22000 raft_consensus.cc:482] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:53:16.611709 22000 raft_consensus.cc:504] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082b64b488b8440badb1ddb7ad263b74" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 42581 } } peers { permanent_uuid: "0dabbab44cec4d378eb643db561fc29a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 46595 } } peers { permanent_uuid: "cfc94c7d3138458bb152b634388a6665" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34615 } }
I20260928 14:53:16.612102 22000 leader_election.cc:288] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 082b64b488b8440badb1ddb7ad263b74 (127.20.116.129:42581), cfc94c7d3138458bb152b634388a6665 (127.20.116.131:34615)
I20260928 14:53:16.616576 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:16.612617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:57647 (local address 127.20.116.129:42581)
0928 14:53:16.612666 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:16.612667 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:16.612677 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:53:16.612712 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:16.612713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:16.612721 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:16.612808 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:16.613032 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.613812 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.615496 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.616117 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.616205 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:16.616213 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:16.616438 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 14:53:16.616503 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:53:16.616564 22002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:16.612458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:34615 (local address 127.20.116.130:60027)
0928 14:53:16.612683 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:16.612691 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:16.612741 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:16.612876 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:16.612876 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:16.612878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:16.612940 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:16.612942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.613937 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.613938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:16.615372 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:16.615376 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.616227 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.616228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:16.616323 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:16.616328 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:53:16.616398 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:53:16.616444 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":69,"thread_start_us":168,"threads_started":1}
I20260928 14:53:16.616618 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:16.612533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:60027 (local address 127.20.116.131:34615)
0928 14:53:16.612585 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:16.612587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:16.612759 (+ 172us) server_negotiation.cc:407] Connection header received
0928 14:53:16.612787 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:16.612788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:16.612798 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:16.612857 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:16.613007 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.613813 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.615534 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.616128 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.616204 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:16.616213 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:16.616438 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 14:53:16.616503 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:53:16.616670 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:16.612351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:42581 (local address 127.20.116.130:57647)
0928 14:53:16.612511 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:16.612524 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:16.612581 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:16.612821 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:16.612822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:16.612824 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:16.612906 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:16.612908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.613937 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.613938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:16.615372 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:16.615376 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.616227 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.616228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:16.616328 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:16.616331 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:53:16.616398 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 14:53:16.616444 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":87}
I20260928 14:53:16.616976 21930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ac96fd73a94cecbf837520e3206564" candidate_uuid: "0dabbab44cec4d378eb643db561fc29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc94c7d3138458bb152b634388a6665" is_pre_election: true
I20260928 14:53:16.617009 21661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ac96fd73a94cecbf837520e3206564" candidate_uuid: "0dabbab44cec4d378eb643db561fc29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "082b64b488b8440badb1ddb7ad263b74" is_pre_election: true
I20260928 14:53:16.617172 21930 raft_consensus.cc:2454] T b9ac96fd73a94cecbf837520e3206564 P cfc94c7d3138458bb152b634388a6665 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dabbab44cec4d378eb643db561fc29a in term 0.
I20260928 14:53:16.617214 21661 raft_consensus.cc:2454] T b9ac96fd73a94cecbf837520e3206564 P 082b64b488b8440badb1ddb7ad263b74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dabbab44cec4d378eb643db561fc29a in term 0.
I20260928 14:53:16.617429 21729 leader_election.cc:302] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dabbab44cec4d378eb643db561fc29a, cfc94c7d3138458bb152b634388a6665; no voters:
I20260928 14:53:16.617571 22000 raft_consensus.cc:2781] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:53:16.617630 22000 raft_consensus.cc:482] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:53:16.617661 22000 raft_consensus.cc:3037] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:16.618368 22000 raft_consensus.cc:504] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082b64b488b8440badb1ddb7ad263b74" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 42581 } } peers { permanent_uuid: "0dabbab44cec4d378eb643db561fc29a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 46595 } } peers { permanent_uuid: "cfc94c7d3138458bb152b634388a6665" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34615 } }
I20260928 14:53:16.618580 22000 leader_election.cc:288] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a [CANDIDATE]: Term 1 election: Requested vote from peers 082b64b488b8440badb1ddb7ad263b74 (127.20.116.129:42581), cfc94c7d3138458bb152b634388a6665 (127.20.116.131:34615)
I20260928 14:53:16.618757 21930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ac96fd73a94cecbf837520e3206564" candidate_uuid: "0dabbab44cec4d378eb643db561fc29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc94c7d3138458bb152b634388a6665"
I20260928 14:53:16.618757 21661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ac96fd73a94cecbf837520e3206564" candidate_uuid: "0dabbab44cec4d378eb643db561fc29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "082b64b488b8440badb1ddb7ad263b74"
I20260928 14:53:16.618834 21661 raft_consensus.cc:3037] T b9ac96fd73a94cecbf837520e3206564 P 082b64b488b8440badb1ddb7ad263b74 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:16.618852 21930 raft_consensus.cc:3037] T b9ac96fd73a94cecbf837520e3206564 P cfc94c7d3138458bb152b634388a6665 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:16.619577 21661 raft_consensus.cc:2454] T b9ac96fd73a94cecbf837520e3206564 P 082b64b488b8440badb1ddb7ad263b74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dabbab44cec4d378eb643db561fc29a in term 1.
I20260928 14:53:16.619580 21930 raft_consensus.cc:2454] T b9ac96fd73a94cecbf837520e3206564 P cfc94c7d3138458bb152b634388a6665 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dabbab44cec4d378eb643db561fc29a in term 1.
I20260928 14:53:16.619751 21729 leader_election.cc:302] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dabbab44cec4d378eb643db561fc29a, cfc94c7d3138458bb152b634388a6665; no voters:
I20260928 14:53:16.619897 22000 raft_consensus.cc:2781] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:16.620085 22000 raft_consensus.cc:686] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a [term 1 LEADER]: Becoming Leader. State: Replica: 0dabbab44cec4d378eb643db561fc29a, State: Running, Role: LEADER
I20260928 14:53:16.620213 22000 consensus_queue.cc:237] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082b64b488b8440badb1ddb7ad263b74" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 42581 } } peers { permanent_uuid: "0dabbab44cec4d378eb643db561fc29a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 46595 } } peers { permanent_uuid: "cfc94c7d3138458bb152b634388a6665" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34615 } }
I20260928 14:53:16.621265 21516 catalog_manager.cc:5495] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dabbab44cec4d378eb643db561fc29a (127.20.116.130). New cstate: current_term: 1 leader_uuid: "0dabbab44cec4d378eb643db561fc29a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082b64b488b8440badb1ddb7ad263b74" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 42581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dabbab44cec4d378eb643db561fc29a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 46595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfc94c7d3138458bb152b634388a6665" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34615 } health_report { overall_health: UNKNOWN } } }
Sep 28 14:53:16 dist-test-slave-pxl6 krb5kdc[21471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.129@KRBTEST.COM
I20260928 14:53:16.638638 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:16.630562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36202 (local address 127.20.116.129:42581)
0928 14:53:16.630631 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:16.630634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:16.630705 (+ 71us) server_negotiation.cc:407] Connection header received
0928 14:53:16.630750 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:16.630751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:16.630762 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:16.630811 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:16.631357 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.631978 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.632906 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.633117 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.633190 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:16.635625 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:16.635638 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:16.635638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:16.635646 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:16.637335 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:16.637770 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:16.637773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:16.637774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:16.637834 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:16.638054 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:16.638055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:16.638055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:16.638224 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:16.638378 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:16.638411 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 14:53:16.638486 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 14:53:16.643209 21843 tablet.cc:2369] T b9ac96fd73a94cecbf837520e3206564 P 0dabbab44cec4d378eb643db561fc29a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 14:53:16 dist-test-slave-pxl6 krb5kdc[21471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607196, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:53:16 dist-test-slave-pxl6 krb5kdc[21471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607196, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.116.129@KRBTEST.COM
I20260928 14:53:16.663149 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:16.655677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36206 (local address 127.20.116.129:42581)
0928 14:53:16.655763 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:16.655766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:16.655867 (+ 101us) server_negotiation.cc:407] Connection header received
0928 14:53:16.655966 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:16.655968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:16.655986 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:16.656046 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:16.656692 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.657201 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.657929 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.658148 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.658229 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:16.660578 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:16.660580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:16.660580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:16.660589 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:16.661837 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:16.662213 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:16.662215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:16.662215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:16.662275 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:16.662485 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:16.662486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:16.662486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:16.662645 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:16.662748 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:16.662936 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 14:53:16.663019 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 14:53:16 dist-test-slave-pxl6 krb5kdc[21471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:53:16 dist-test-slave-pxl6 krb5kdc[21471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.129@KRBTEST.COM
I20260928 14:53:16.686739 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:16.678969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36220 (local address 127.20.116.129:42581)
0928 14:53:16.679069 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:16.679075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:16.679150 (+ 75us) server_negotiation.cc:407] Connection header received
0928 14:53:16.679205 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:16.679206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:16.679218 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:16.679278 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:16.679904 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.680395 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.681239 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:16.681436 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:16.681561 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:16.684022 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:16.684024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:16.684024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:16.684033 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:16.685490 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:16.685849 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:16.685850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:16.685850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:16.685900 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:16.686133 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:16.686133 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:16.686133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:16.686227 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:16.686350 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:16.686467 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 14:53:16.686630 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 14:53:16.687259 21621 tablet_service.cc:2908] Rejecting scan request for tablet b9ac96fd73a94cecbf837520e3206564: Uninitialized: clean time has not yet been initialized
I20260928 14:53:16.688127 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 21578
I20260928 14:53:16.693439 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 21712
I20260928 14:53:16.698508 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 21847
I20260928 14:53:16.703585 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 21484
2026-09-28T14:53:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1975 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:53:16 dist-test-slave-pxl6 krb5kdc[22017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:53:16 dist-test-slave-pxl6 krb5kdc[22017](info): set up 2 sockets
Sep 28 14:53:16 dist-test-slave-pxl6 krb5kdc[22017](info): commencing operation
krb5kdc: starting...
W20260928 14:53:17.755074 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:53:17 dist-test-slave-pxl6 krb5kdc[22017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:53:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:53:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:53:17.877560 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:41203
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:41203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:17.960527 22030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:17.960758 22030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:17.960783 22030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:17.961894 22030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:53:17.961928 22030 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 14:53:17.961941 22030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:17.961954 22030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:17.961967 22030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:53:17.961978 22030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:53:17.963481 22030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:41203
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:41203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:17.963690 22030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:17.963862 22030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:53:17.969111 22030 server_base.cc:1034] running on GCE node
W20260928 14:53:17.969017 22038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:17.969017 22035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:17.969079 22036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:17.969501 22030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:17.969779 22030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:17.970950 22030 hybrid_clock.cc:648] HybridClock initialized: now 1790607197970919 us; error 58 us; skew 500 ppm
Sep 28 14:53:17 dist-test-slave-pxl6 krb5kdc[22017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607197, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:17.977772 22030 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:53:17.978279 22030 webserver.cc:460] Webserver started at http://127.20.116.190:41327/ using document root <none> and password file <none>
I20260928 14:53:17.978515 22030 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:17.978609 22030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:17.978806 22030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:17.979912 22030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "50c60f6555694292b065664094fae6ec"
format_stamp: "Formatted at 2026-09-28 14:53:17 on dist-test-slave-pxl6"
server_key: "da61b546ee433b13bd6b9b1f31172d1c"
server_key_iv: "a54f3f6cc3f2ecd38610bf01e70350e6"
server_key_version: "encryptionkey@0"
I20260928 14:53:17.980288 22030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "50c60f6555694292b065664094fae6ec"
format_stamp: "Formatted at 2026-09-28 14:53:17 on dist-test-slave-pxl6"
server_key: "da61b546ee433b13bd6b9b1f31172d1c"
server_key_iv: "a54f3f6cc3f2ecd38610bf01e70350e6"
server_key_version: "encryptionkey@0"
I20260928 14:53:17.981676 22030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:53:17.982442 22045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:17.982662 22030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:17.982779 22030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "50c60f6555694292b065664094fae6ec"
format_stamp: "Formatted at 2026-09-28 14:53:17 on dist-test-slave-pxl6"
server_key: "da61b546ee433b13bd6b9b1f31172d1c"
server_key_iv: "a54f3f6cc3f2ecd38610bf01e70350e6"
server_key_version: "encryptionkey@0"
I20260928 14:53:17.982863 22030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:18.003222 22030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:18.005677 22030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:18.005846 22030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:18.011065 22030 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:41203
I20260928 14:53:18.011159 22107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:41203 every 8 connection(s)
I20260928 14:53:18.011561 22030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:53:18.012642 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 22030
I20260928 14:53:18.012738 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:53:18.012928 20946 external_mini_cluster.cc:1428] Setting key f04b9f6cc46911399741b1351b3d0736
I20260928 14:53:18.013187 22108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:18.017192 22108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec: Bootstrap starting.
I20260928 14:53:18.017910 22108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:18.018244 22108 log.cc:824] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:18.019086 22108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec: No bootstrap required, opened a new log
Sep 28 14:53:18 dist-test-slave-pxl6 krb5kdc[22017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:18.021112 22108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c60f6555694292b065664094fae6ec" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 41203 } }
I20260928 14:53:18.021276 22108 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:18.021325 22108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50c60f6555694292b065664094fae6ec, State: Initialized, Role: FOLLOWER
I20260928 14:53:18.021464 22108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c60f6555694292b065664094fae6ec" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 41203 } }
I20260928 14:53:18.021564 22108 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:53:18.021598 22108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:53:18.021669 22108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:18.022393 22108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c60f6555694292b065664094fae6ec" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 41203 } }
I20260928 14:53:18.022488 22108 leader_election.cc:302] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50c60f6555694292b065664094fae6ec; no voters:
I20260928 14:53:18.022748 22108 leader_election.cc:288] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:53:18.022780 22113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:18.022902 22113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec [term 1 LEADER]: Becoming Leader. State: Replica: 50c60f6555694292b065664094fae6ec, State: Running, Role: LEADER
I20260928 14:53:18.022989 22113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c60f6555694292b065664094fae6ec" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 41203 } }
I20260928 14:53:18.023038 22108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:18.023463 22114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50c60f6555694292b065664094fae6ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c60f6555694292b065664094fae6ec" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 41203 } } }
I20260928 14:53:18.023554 22114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:18.023842 22115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50c60f6555694292b065664094fae6ec. Latest consensus state: current_term: 1 leader_uuid: "50c60f6555694292b065664094fae6ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c60f6555694292b065664094fae6ec" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 41203 } } }
I20260928 14:53:18.023993 22115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:18.024233 22122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:53:18.024799 22122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:53:18.025307 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.013943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59694 (local address 127.20.116.190:41203)
0928 14:53:18.014134 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:18.014146 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:18.014181 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:53:18.014823 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:18.014833 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:18.014914 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:18.015073 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:18.015667 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.017059 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.017888 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.018559 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.018671 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.021170 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:18.021174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:18.021174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:18.021186 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:18.023407 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:18.023890 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:18.023894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:18.023894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:18.024003 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:18.024290 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:18.024291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:18.024291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:18.024645 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:18.024829 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:18.024941 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 14:53:18.025085 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":106,"threads_started":1}
I20260928 14:53:18.026768 22122 catalog_manager.cc:1347] Generated new cluster ID: 972bbba63fd44b91be7e5e36c2bfac40
I20260928 14:53:18.026830 22122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:53:18.047667 22122 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:53:18.048287 22122 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:53:18.050568 22122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec: Generated new TSK 0
I20260928 14:53:18.050810 22122 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
I20260928 14:53:18.083966 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:41203
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:18.166056 22134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:18.166309 22134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:18.166348 22134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:18.167570 22134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:18.167644 22134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:18.167680 22134 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:53:18.167721 22134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:53:18.169193 22134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.116.190:41203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:18.169474 22134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:18.169713 22134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:18.175222 22142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:18.175222 22139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:18.175490 22134 server_base.cc:1034] running on GCE node
W20260928 14:53:18.175267 22140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:18.175786 22134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:18.176043 22134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:18.177237 22134 hybrid_clock.cc:648] HybridClock initialized: now 1790607198177216 us; error 37 us; skew 500 ppm
Sep 28 14:53:18 dist-test-slave-pxl6 krb5kdc[22017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607198, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:18.179798 22134 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:53:18.180186 22134 webserver.cc:460] Webserver started at http://127.20.116.129:41577/ using document root <none> and password file <none>
I20260928 14:53:18.180384 22134 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:18.180424 22134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:18.180521 22134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:18.181540 22134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "86df19358ca742489cce32be42a6788e"
format_stamp: "Formatted at 2026-09-28 14:53:18 on dist-test-slave-pxl6"
server_key: "2aa1e0cfa88e8f33e8e2dfac6d82ca54"
server_key_iv: "32cc06d010cd9a352ce0a9693032f778"
server_key_version: "encryptionkey@0"
I20260928 14:53:18.181859 22134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "86df19358ca742489cce32be42a6788e"
format_stamp: "Formatted at 2026-09-28 14:53:18 on dist-test-slave-pxl6"
server_key: "2aa1e0cfa88e8f33e8e2dfac6d82ca54"
server_key_iv: "32cc06d010cd9a352ce0a9693032f778"
server_key_version: "encryptionkey@0"
I20260928 14:53:18.183192 22134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:53:18.183874 22149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:18.184047 22134 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:53:18.184145 22134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "86df19358ca742489cce32be42a6788e"
format_stamp: "Formatted at 2026-09-28 14:53:18 on dist-test-slave-pxl6"
server_key: "2aa1e0cfa88e8f33e8e2dfac6d82ca54"
server_key_iv: "32cc06d010cd9a352ce0a9693032f778"
server_key_version: "encryptionkey@0"
I20260928 14:53:18.184201 22134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:18.195838 22134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:18.198081 22134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:18.198199 22134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:18.198701 22134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:18.198743 22134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:18.198765 22134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:18.198778 22134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:18.205433 22134 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:41677
I20260928 14:53:18.205515 22264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:41677 every 8 connection(s)
I20260928 14:53:18.205797 22134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:53:18 dist-test-slave-pxl6 krb5kdc[22017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607198, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:18.209474 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 22134
I20260928 14:53:18.209610 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:53:18.209782 20946 external_mini_cluster.cc:1428] Setting key 008bcae582a4a519c2c8f58647a8e07e
I20260928 14:53:18.209807 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.199136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:41203 (local address 127.20.116.129:57741)
0928 14:53:18.199303 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:18.199343 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:18.199829 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:18.200108 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:18.200113 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:18.200556 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:18.200851 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.200859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.201913 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.201914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.203410 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.203413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.203802 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.203804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.203977 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.204599 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:18.204612 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:18.206358 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:18.208088 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:18.208096 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:18.208098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:18.208321 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:18.208540 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:18.208541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:18.208541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:18.208672 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:18.209021 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:18.209028 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:18.209242 (+ 214us) client_negotiation.cc:769] Sending connection context
0928 14:53:18.209384 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 14:53:18.209565 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
I20260928 14:53:18.210173 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.199174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:57741 (local address 127.20.116.190:41203)
0928 14:53:18.199239 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:18.199242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:18.199419 (+ 177us) server_negotiation.cc:407] Connection header received
0928 14:53:18.199957 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:18.199959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:18.199972 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:18.200015 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:18.200942 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.201797 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.203510 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.203715 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.203773 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.206460 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:18.206462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:18.206463 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:18.206472 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:18.207993 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:18.208402 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:18.208403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:18.208404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:18.208469 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:18.208770 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:18.208772 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:18.208772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:18.208938 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:18.209021 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:18.209441 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 14:53:18.209776 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:53:18.210695 22265 heartbeater.cc:347] Connected to a master server at 127.20.116.190:41203
I20260928 14:53:18.210801 22265 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:18.210985 22265 heartbeater.cc:511] Master 127.20.116.190:41203 requested a full tablet report, sending...
I20260928 14:53:18.211509 22062 ts_manager.cc:194] Registered new tserver with Master: 86df19358ca742489cce32be42a6788e (127.20.116.129:41677)
I20260928 14:53:18.212252 22062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:57741
WARNING: no policy specified for kudu/127.20.116.130@KRBTEST.COM; defaulting to no policy
I20260928 14:53:18.218289 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.211814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:54423 (local address 127.20.116.190:41203)
0928 14:53:18.212469 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:18.212471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:18.212616 (+ 145us) server_negotiation.cc:407] Connection header received
0928 14:53:18.212726 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:18.212728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:18.212742 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:18.212806 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:18.213149 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.213646 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.214430 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.214712 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.214796 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.215725 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:18.215727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:18.215727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:18.215734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:18.217144 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:18.217491 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:18.217492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:18.217492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:18.217545 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:18.217760 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:18.217761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:18.217762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:18.217875 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:18.217953 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:18.218124 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 14:53:18.218200 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635}
I20260928 14:53:18.218299 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.211798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:41203 (local address 127.20.116.129:54423)
0928 14:53:18.212543 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:18.212554 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:18.212638 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:18.212817 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:18.212818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:18.213010 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:18.213065 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.213068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.213743 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.213744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.214349 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.214351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.214798 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.214799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.214875 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.215324 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:18.215329 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:18.215619 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:18.217245 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:18.217246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:18.217246 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:18.217418 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:18.217625 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:18.217626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:18.217626 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:18.217685 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:18.217945 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:18.217946 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:18.217992 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 14:53:18.218096 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 14:53:18.218201 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":80,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
I20260928 14:53:18.235621 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:41203
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:18.321748 22274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:18.322001 22274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:18.322041 22274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:18.323408 22274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:18.323468 22274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:18.323505 22274 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:53:18.323562 22274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:53:18.325202 22274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.116.190:41203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:18.325475 22274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:18.325718 22274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:18.330885 22282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:18.330958 22279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:18.330871 22280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:18.331092 22274 server_base.cc:1034] running on GCE node
I20260928 14:53:18.331276 22274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:18.331477 22274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:18.332623 22274 hybrid_clock.cc:648] HybridClock initialized: now 1790607198332608 us; error 27 us; skew 500 ppm
Sep 28 14:53:18 dist-test-slave-pxl6 krb5kdc[22017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607198, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:18.335191 22274 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:53:18.335575 22274 webserver.cc:460] Webserver started at http://127.20.116.130:39275/ using document root <none> and password file <none>
I20260928 14:53:18.335774 22274 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:18.335814 22274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:18.335901 22274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:18.336969 22274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "e70bc98543884c059729cebbd0010cf9"
format_stamp: "Formatted at 2026-09-28 14:53:18 on dist-test-slave-pxl6"
server_key: "663e0f3876dc5f09621444f51be67fcd"
server_key_iv: "9afefca3969ecbaf9305d6f3a225ea73"
server_key_version: "encryptionkey@0"
I20260928 14:53:18.337273 22274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "e70bc98543884c059729cebbd0010cf9"
format_stamp: "Formatted at 2026-09-28 14:53:18 on dist-test-slave-pxl6"
server_key: "663e0f3876dc5f09621444f51be67fcd"
server_key_iv: "9afefca3969ecbaf9305d6f3a225ea73"
server_key_version: "encryptionkey@0"
I20260928 14:53:18.338696 22274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 14:53:18.339511 22289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:18.339707 22274 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:18.339797 22274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "e70bc98543884c059729cebbd0010cf9"
format_stamp: "Formatted at 2026-09-28 14:53:18 on dist-test-slave-pxl6"
server_key: "663e0f3876dc5f09621444f51be67fcd"
server_key_iv: "9afefca3969ecbaf9305d6f3a225ea73"
server_key_version: "encryptionkey@0"
I20260928 14:53:18.339852 22274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:18.362478 22274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:18.364631 22274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:18.364761 22274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:18.365355 22274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:18.365398 22274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:18.365447 22274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:18.365497 22274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:18.372840 22274 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:44025
I20260928 14:53:18.373243 22274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:53:18 dist-test-slave-pxl6 krb5kdc[22017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607198, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:18.373508 22404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:44025 every 8 connection(s)
I20260928 14:53:18.376540 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.366215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:44453 (local address 127.20.116.190:41203)
0928 14:53:18.366295 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:18.366298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:18.366503 (+ 205us) server_negotiation.cc:407] Connection header received
0928 14:53:18.367227 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:18.367229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:18.367247 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:18.367336 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:18.368247 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.369133 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.370898 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.371104 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.371200 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.373952 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:18.373954 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:18.373954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:18.373961 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:18.375260 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:18.375663 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:18.375664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:18.375665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:18.375721 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:18.375947 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:18.375948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:18.375948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:18.376078 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:18.376141 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:18.376392 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 14:53:18.376463 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:53:18.376736 22298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.366172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:41203 (local address 127.20.116.130:44453)
0928 14:53:18.366362 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:18.366405 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:18.367098 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:18.367436 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:18.367441 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:18.367828 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:18.368130 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.368136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.369248 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.369249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.370802 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.370806 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.371211 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.371212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.371379 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.372015 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:18.372028 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:18.373857 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:18.375334 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:18.375341 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:18.375342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:18.375574 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:18.375789 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:18.375789 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:18.375790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:18.375889 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:18.376140 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:18.376145 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:18.376278 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 14:53:18.376364 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 14:53:18.376498 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":115,"threads_started":1}
I20260928 14:53:18.377242 22405 heartbeater.cc:347] Connected to a master server at 127.20.116.190:41203
I20260928 14:53:18.377331 22405 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:18.377548 22405 heartbeater.cc:511] Master 127.20.116.190:41203 requested a full tablet report, sending...
I20260928 14:53:18.377957 22061 ts_manager.cc:194] Registered new tserver with Master: e70bc98543884c059729cebbd0010cf9 (127.20.116.130:44025)
I20260928 14:53:18.378309 22061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:44453
I20260928 14:53:18.380959 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 22274
I20260928 14:53:18.381033 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:53:18.381178 20946 external_mini_cluster.cc:1428] Setting key 4c1425125cf67523483e6edf31cc55e7
I20260928 14:53:18.384648 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.378425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:41783 (local address 127.20.116.190:41203)
0928 14:53:18.378465 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:18.378467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:18.378673 (+ 206us) server_negotiation.cc:407] Connection header received
0928 14:53:18.378747 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:18.378748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:18.378757 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:18.378842 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:18.379198 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.379682 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.380379 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.380635 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.380723 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.381817 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:18.381819 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:18.381819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:18.381827 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:18.383468 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:18.383845 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:18.383847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:18.383847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:18.383919 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:18.384147 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:18.384147 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:18.384148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:18.384287 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:18.384341 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:18.384515 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 14:53:18.384578 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:53:18.384742 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.378435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:41203 (local address 127.20.116.130:41783)
0928 14:53:18.378595 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:18.378606 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:18.378686 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:18.378848 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:18.378849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:18.379023 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:18.379117 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.379118 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.379777 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.379778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.380293 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.380295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.380726 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.380727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.380821 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.381290 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:18.381295 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:18.381728 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:18.383555 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:18.383555 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:18.383556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:18.383758 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:18.383993 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:18.383994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:18.383994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:18.384068 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:18.384351 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:18.384353 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:18.384418 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 14:53:18.384499 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 14:53:18.384612 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":2,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for kudu/127.20.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
I20260928 14:53:18.410352 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:41203
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:18.492884 22414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:18.493098 22414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:18.493122 22414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:18.494313 22414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:18.494359 22414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:18.494379 22414 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:53:18.494397 22414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:53:18.495963 22414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.116.190:41203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:18.496234 22414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:18.496484 22414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:18.501750 22419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:18.501750 22422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:18.501770 22414 server_base.cc:1034] running on GCE node
W20260928 14:53:18.501745 22420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:18.502151 22414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:18.502352 22414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:18.503494 22414 hybrid_clock.cc:648] HybridClock initialized: now 1790607198503473 us; error 42 us; skew 500 ppm
Sep 28 14:53:18 dist-test-slave-pxl6 krb5kdc[22017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607198, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:18.506126 22414 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:53:18.506624 22414 webserver.cc:460] Webserver started at http://127.20.116.131:34417/ using document root <none> and password file <none>
I20260928 14:53:18.506868 22414 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:18.506933 22414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:18.507085 22414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:18.508127 22414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "3851b749dbbb4d548f57e50735ed8bca"
format_stamp: "Formatted at 2026-09-28 14:53:18 on dist-test-slave-pxl6"
server_key: "e8ca011b9c4f836d0805f5ff47f520d3"
server_key_iv: "74fa46437e8a79332887d728203be185"
server_key_version: "encryptionkey@0"
I20260928 14:53:18.508484 22414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "3851b749dbbb4d548f57e50735ed8bca"
format_stamp: "Formatted at 2026-09-28 14:53:18 on dist-test-slave-pxl6"
server_key: "e8ca011b9c4f836d0805f5ff47f520d3"
server_key_iv: "74fa46437e8a79332887d728203be185"
server_key_version: "encryptionkey@0"
I20260928 14:53:18.509790 22414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:53:18.510598 22429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:18.510804 22414 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:53:18.510912 22414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "3851b749dbbb4d548f57e50735ed8bca"
format_stamp: "Formatted at 2026-09-28 14:53:18 on dist-test-slave-pxl6"
server_key: "e8ca011b9c4f836d0805f5ff47f520d3"
server_key_iv: "74fa46437e8a79332887d728203be185"
server_key_version: "encryptionkey@0"
I20260928 14:53:18.510998 22414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:18.556941 22414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:18.559456 22414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:18.559628 22414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:18.560200 22414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:18.560262 22414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:18.560326 22414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:18.560356 22414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:53:18 dist-test-slave-pxl6 krb5kdc[22017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607198, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:18.567405 22414 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:38191
I20260928 14:53:18.567523 22544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:38191 every 8 connection(s)
I20260928 14:53:18.567858 22414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:53:18.570420 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.560736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:56789 (local address 127.20.116.190:41203)
0928 14:53:18.560814 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:18.560816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:18.561050 (+ 234us) server_negotiation.cc:407] Connection header received
0928 14:53:18.561603 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:18.561605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:18.561620 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:18.561709 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:18.562645 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.563500 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.564608 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.564942 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.565018 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.567712 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:18.567714 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:18.567714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:18.567723 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:18.569274 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:18.569655 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:18.569656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:18.569656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:18.569710 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:18.569894 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:18.569895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:18.569895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:18.570001 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:18.570051 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:18.570292 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 14:53:18.570356 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:53:18.570669 22438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.560702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:41203 (local address 127.20.116.131:56789)
0928 14:53:18.560896 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:18.560943 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:18.561463 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:18.561771 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:18.561774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:18.562184 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:18.562490 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.562499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.563615 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.563616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.564508 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.564510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.565037 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.565038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.565196 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.565842 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:18.565858 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:18.567613 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:18.569349 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:18.569354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:18.569356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:18.569568 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:18.569754 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:18.569754 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:18.569755 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:18.569841 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:18.570057 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:18.570062 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:18.570178 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 14:53:18.570261 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 14:53:18.570387 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260928 14:53:18.571194 22545 heartbeater.cc:347] Connected to a master server at 127.20.116.190:41203
I20260928 14:53:18.571276 22545 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:18.571486 22545 heartbeater.cc:511] Master 127.20.116.190:41203 requested a full tablet report, sending...
I20260928 14:53:18.571890 22062 ts_manager.cc:194] Registered new tserver with Master: 3851b749dbbb4d548f57e50735ed8bca (127.20.116.131:38191)
I20260928 14:53:18.572458 22062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:56789
I20260928 14:53:18.575858 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 22414
I20260928 14:53:18.575958 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:53:18.576161 20946 external_mini_cluster.cc:1428] Setting key c2e02b31b665a947222fdfd56ddf0af9
I20260928 14:53:18.577219 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:53:18.578478 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.572305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:48253 (local address 127.20.116.190:41203)
0928 14:53:18.572594 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:18.572596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:18.572604 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:53:18.572622 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:18.572623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:18.572632 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:18.572705 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:18.573074 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.573540 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.574251 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.574460 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.574537 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.575548 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:18.575550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:18.575550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:18.575557 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:18.577274 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:18.577666 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:18.577667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:18.577667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:18.577727 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:18.577938 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:18.577938 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:18.577939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:18.578048 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:18.578133 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:18.578325 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 14:53:18.578415 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269}
I20260928 14:53:18.578634 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.572253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:41203 (local address 127.20.116.131:48253)
0928 14:53:18.572390 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:18.572399 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:18.572443 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:18.572726 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:18.572728 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:18.572909 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:18.572978 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.572979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.573632 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.573632 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.574167 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.574169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.574578 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.574579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.574659 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.575150 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:18.575155 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:18.575436 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:18.577368 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:18.577369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:18.577380 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:18.577594 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:18.577800 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:18.577801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:18.577802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:18.577875 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:18.578140 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:18.578140 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:18.578193 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 14:53:18.578263 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:53:18.578428 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":7,"thread_start_us":82,"threads_started":1}
Sep 28 14:53:18 dist-test-slave-pxl6 krb5kdc[22017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607198, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:53:18 dist-test-slave-pxl6 krb5kdc[22017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607198, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:18.601339 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.593912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43394 (local address 127.20.116.190:41203)
0928 14:53:18.593968 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:18.593970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:18.593981 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:53:18.594095 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:18.594096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:18.594106 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:18.594166 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:18.594819 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.595334 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.596116 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.596316 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.596403 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.598698 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:18.598700 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:18.598700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:18.598708 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:18.600107 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:18.600479 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:18.600480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:18.600481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:18.600539 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:18.600779 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:18.600780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:18.600780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:18.600917 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:18.601028 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:18.601148 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 14:53:18.601229 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 14:53:18.603240 22062 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:53:18.604976 22062 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:53:18.611984 22564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.607542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:41677 (local address 127.0.0.1:32958)
0928 14:53:18.607839 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:18.607887 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:18.607962 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:18.608162 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:18.608165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:18.608168 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:18.608278 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.608280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.609416 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.609417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.610686 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.610689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.611641 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.611642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.611730 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.611749 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 14:53:18.611818 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 14:53:18.611883 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":66,"threads_started":1}
I20260928 14:53:18.612090 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.607542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:38191 (local address 127.0.0.1:47366)
0928 14:53:18.608061 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:18.608069 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:18.608176 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:18.608437 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:18.608439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:18.608441 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:18.608498 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.608499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.609787 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.609788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.610913 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.610916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.611777 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.611778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.611855 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.611860 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:53:18.611919 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 14:53:18.611963 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":4,"thread_start_us":77,"threads_started":1}
I20260928 14:53:18.612095 22565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.607723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47366 (local address 127.20.116.131:38191)
0928 14:53:18.607917 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:18.607922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:18.608143 (+ 221us) server_negotiation.cc:407] Connection header received
0928 14:53:18.608269 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:18.608272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:18.608289 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:18.608343 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:18.608571 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.609703 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.610997 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.611684 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.611758 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.611811 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:18.611944 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 14:53:18.611983 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
I20260928 14:53:18.612388 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.607774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32958 (local address 127.20.116.129:41677)
0928 14:53:18.607980 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:18.607985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:18.607996 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:53:18.608044 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:18.608046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:18.608060 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:18.608149 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:18.608375 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.609334 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.610781 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.611533 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.611622 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.611679 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:18.612213 (+ 534us) server_negotiation.cc:299] Negotiation successful
0928 14:53:18.612265 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":85,"threads_started":1}
I20260928 14:53:18.612432 22563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.607547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:44025 (local address 127.0.0.1:33572)
0928 14:53:18.608602 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:18.608613 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:18.608663 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:18.608841 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:18.608841 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:18.608844 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:18.608902 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.608903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.610058 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.610059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.611183 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.611187 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.612140 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.612141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.612276 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.612281 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:53:18.612330 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 14:53:18.612368 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"thread_start_us":125,"threads_started":1}
I20260928 14:53:18.612922 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.607829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.20.116.130:44025)
0928 14:53:18.607999 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:18.608003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:18.608678 (+ 675us) server_negotiation.cc:407] Connection header received
0928 14:53:18.608727 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:18.608730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:18.608745 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:18.608831 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:18.608966 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.609997 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.611271 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.611975 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.612052 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.612110 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:18.612564 (+ 454us) server_negotiation.cc:299] Negotiation successful
0928 14:53:18.612608 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":74,"threads_started":1}
I20260928 14:53:18.612866 22479 tablet_service.cc:1467] Processing CreateTablet for tablet cad47beffa4447a69d5d7cb1055ed3b3 (DEFAULT_TABLE table=test-table [id=ad425fa2b6ce43f6acf7261a680fb82c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:18.613027 22199 tablet_service.cc:1467] Processing CreateTablet for tablet cad47beffa4447a69d5d7cb1055ed3b3 (DEFAULT_TABLE table=test-table [id=ad425fa2b6ce43f6acf7261a680fb82c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:18.613286 22479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cad47beffa4447a69d5d7cb1055ed3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:18.613407 22199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cad47beffa4447a69d5d7cb1055ed3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:18.616135 22568 tablet_bootstrap.cc:492] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca: Bootstrap starting.
I20260928 14:53:18.616490 22569 tablet_bootstrap.cc:492] T cad47beffa4447a69d5d7cb1055ed3b3 P 86df19358ca742489cce32be42a6788e: Bootstrap starting.
I20260928 14:53:18.616694 22339 tablet_service.cc:1467] Processing CreateTablet for tablet cad47beffa4447a69d5d7cb1055ed3b3 (DEFAULT_TABLE table=test-table [id=ad425fa2b6ce43f6acf7261a680fb82c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:18.617058 22339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cad47beffa4447a69d5d7cb1055ed3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:18.617153 22568 tablet_bootstrap.cc:654] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:18.617202 22569 tablet_bootstrap.cc:654] T cad47beffa4447a69d5d7cb1055ed3b3 P 86df19358ca742489cce32be42a6788e: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:18.617480 22569 log.cc:824] T cad47beffa4447a69d5d7cb1055ed3b3 P 86df19358ca742489cce32be42a6788e: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:18.617473 22568 log.cc:824] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:18.618197 22568 tablet_bootstrap.cc:492] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca: No bootstrap required, opened a new log
I20260928 14:53:18.618247 22569 tablet_bootstrap.cc:492] T cad47beffa4447a69d5d7cb1055ed3b3 P 86df19358ca742489cce32be42a6788e: No bootstrap required, opened a new log
I20260928 14:53:18.618278 22568 ts_tablet_manager.cc:1397] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:53:18.618317 22569 ts_tablet_manager.cc:1397] T cad47beffa4447a69d5d7cb1055ed3b3 P 86df19358ca742489cce32be42a6788e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:53:18.619917 22568 raft_consensus.cc:348] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } }
I20260928 14:53:18.620069 22568 raft_consensus.cc:374] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:18.620113 22568 raft_consensus.cc:729] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3851b749dbbb4d548f57e50735ed8bca, State: Initialized, Role: FOLLOWER
I20260928 14:53:18.620229 22568 consensus_queue.cc:260] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } }
I20260928 14:53:18.620496 22545 heartbeater.cc:503] Master 127.20.116.190:41203 was elected leader, sending a full tablet report...
I20260928 14:53:18.620445 22569 raft_consensus.cc:348] T cad47beffa4447a69d5d7cb1055ed3b3 P 86df19358ca742489cce32be42a6788e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } }
I20260928 14:53:18.620607 22569 raft_consensus.cc:374] T cad47beffa4447a69d5d7cb1055ed3b3 P 86df19358ca742489cce32be42a6788e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:18.620654 22569 raft_consensus.cc:729] T cad47beffa4447a69d5d7cb1055ed3b3 P 86df19358ca742489cce32be42a6788e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86df19358ca742489cce32be42a6788e, State: Initialized, Role: FOLLOWER
I20260928 14:53:18.620494 22568 ts_tablet_manager.cc:1428] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:53:18.620795 22569 consensus_queue.cc:260] T cad47beffa4447a69d5d7cb1055ed3b3 P 86df19358ca742489cce32be42a6788e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } }
I20260928 14:53:18.621088 22573 tablet_bootstrap.cc:492] T cad47beffa4447a69d5d7cb1055ed3b3 P e70bc98543884c059729cebbd0010cf9: Bootstrap starting.
I20260928 14:53:18.621086 22265 heartbeater.cc:503] Master 127.20.116.190:41203 was elected leader, sending a full tablet report...
I20260928 14:53:18.621078 22569 ts_tablet_manager.cc:1428] T cad47beffa4447a69d5d7cb1055ed3b3 P 86df19358ca742489cce32be42a6788e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:53:18.621901 22573 tablet_bootstrap.cc:654] T cad47beffa4447a69d5d7cb1055ed3b3 P e70bc98543884c059729cebbd0010cf9: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:18.622247 22573 log.cc:824] T cad47beffa4447a69d5d7cb1055ed3b3 P e70bc98543884c059729cebbd0010cf9: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:18.622980 22573 tablet_bootstrap.cc:492] T cad47beffa4447a69d5d7cb1055ed3b3 P e70bc98543884c059729cebbd0010cf9: No bootstrap required, opened a new log
I20260928 14:53:18.623061 22573 ts_tablet_manager.cc:1397] T cad47beffa4447a69d5d7cb1055ed3b3 P e70bc98543884c059729cebbd0010cf9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:53:18.624655 22573 raft_consensus.cc:348] T cad47beffa4447a69d5d7cb1055ed3b3 P e70bc98543884c059729cebbd0010cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } }
I20260928 14:53:18.624804 22573 raft_consensus.cc:374] T cad47beffa4447a69d5d7cb1055ed3b3 P e70bc98543884c059729cebbd0010cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:18.624846 22573 raft_consensus.cc:729] T cad47beffa4447a69d5d7cb1055ed3b3 P e70bc98543884c059729cebbd0010cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e70bc98543884c059729cebbd0010cf9, State: Initialized, Role: FOLLOWER
I20260928 14:53:18.624964 22573 consensus_queue.cc:260] T cad47beffa4447a69d5d7cb1055ed3b3 P e70bc98543884c059729cebbd0010cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } }
I20260928 14:53:18.625212 22573 ts_tablet_manager.cc:1428] T cad47beffa4447a69d5d7cb1055ed3b3 P e70bc98543884c059729cebbd0010cf9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:53:18.625299 22405 heartbeater.cc:503] Master 127.20.116.190:41203 was elected leader, sending a full tablet report...
W20260928 14:53:18.706681 22266 tablet.cc:2369] T cad47beffa4447a69d5d7cb1055ed3b3 P 86df19358ca742489cce32be42a6788e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:53:18.710134 22572 raft_consensus.cc:482] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:53:18.710317 22572 raft_consensus.cc:504] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } }
I20260928 14:53:18.710790 22572 leader_election.cc:288] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86df19358ca742489cce32be42a6788e (127.20.116.129:41677), e70bc98543884c059729cebbd0010cf9 (127.20.116.130:44025)
I20260928 14:53:18.714125 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.711305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:59901 (local address 127.20.116.129:41677)
0928 14:53:18.711372 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:18.711374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:18.711382 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:53:18.711407 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:18.711408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:18.711416 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:18.711459 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:18.711708 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.712408 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.713357 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.713769 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.713825 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.713832 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:18.714008 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 14:53:18.714054 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:53:18.714184 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.711135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:45837 (local address 127.20.116.130:44025)
0928 14:53:18.711183 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:18.711186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:18.711222 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:53:18.711253 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:18.711254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:18.711264 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:18.711651 (+ 387us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:18.711694 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.712240 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.713347 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.713814 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.713887 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.713895 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:18.714073 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 14:53:18.714117 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:53:18.714157 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.710974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:41677 (local address 127.20.116.131:59901)
0928 14:53:18.711179 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:18.711190 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:18.711247 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:18.711515 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:18.711515 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:18.711517 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:18.711636 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.711638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.712489 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.712491 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.713252 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.713254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.713842 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.713844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.713928 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.713932 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:53:18.713984 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 14:53:18.714044 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":50,"thread_start_us":119,"threads_started":1}
I20260928 14:53:18.714175 22438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.710969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:44025 (local address 127.20.116.131:45837)
0928 14:53:18.711086 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:18.711099 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:18.711145 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:18.711415 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:18.711416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:18.711418 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:18.711575 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.711585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.712322 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.712323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.713240 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.713243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.713893 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.713894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.713984 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.713988 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:53:18.714046 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 14:53:18.714080 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260928 14:53:18.714619 22219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cad47beffa4447a69d5d7cb1055ed3b3" candidate_uuid: "3851b749dbbb4d548f57e50735ed8bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86df19358ca742489cce32be42a6788e" is_pre_election: true
I20260928 14:53:18.714694 22359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cad47beffa4447a69d5d7cb1055ed3b3" candidate_uuid: "3851b749dbbb4d548f57e50735ed8bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e70bc98543884c059729cebbd0010cf9" is_pre_election: true
I20260928 14:53:18.714795 22219 raft_consensus.cc:2454] T cad47beffa4447a69d5d7cb1055ed3b3 P 86df19358ca742489cce32be42a6788e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3851b749dbbb4d548f57e50735ed8bca in term 0.
I20260928 14:53:18.714848 22359 raft_consensus.cc:2454] T cad47beffa4447a69d5d7cb1055ed3b3 P e70bc98543884c059729cebbd0010cf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3851b749dbbb4d548f57e50735ed8bca in term 0.
I20260928 14:53:18.715016 22432 leader_election.cc:302] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3851b749dbbb4d548f57e50735ed8bca, 86df19358ca742489cce32be42a6788e; no voters:
I20260928 14:53:18.715137 22572 raft_consensus.cc:2781] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:53:18.715196 22572 raft_consensus.cc:482] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:53:18.715225 22572 raft_consensus.cc:3037] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:18.716193 22572 raft_consensus.cc:504] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } }
I20260928 14:53:18.716341 22572 leader_election.cc:288] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca [CANDIDATE]: Term 1 election: Requested vote from peers 86df19358ca742489cce32be42a6788e (127.20.116.129:41677), e70bc98543884c059729cebbd0010cf9 (127.20.116.130:44025)
I20260928 14:53:18.716506 22219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cad47beffa4447a69d5d7cb1055ed3b3" candidate_uuid: "3851b749dbbb4d548f57e50735ed8bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86df19358ca742489cce32be42a6788e"
I20260928 14:53:18.716528 22359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cad47beffa4447a69d5d7cb1055ed3b3" candidate_uuid: "3851b749dbbb4d548f57e50735ed8bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e70bc98543884c059729cebbd0010cf9"
I20260928 14:53:18.716596 22219 raft_consensus.cc:3037] T cad47beffa4447a69d5d7cb1055ed3b3 P 86df19358ca742489cce32be42a6788e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:18.716617 22359 raft_consensus.cc:3037] T cad47beffa4447a69d5d7cb1055ed3b3 P e70bc98543884c059729cebbd0010cf9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:18.717312 22219 raft_consensus.cc:2454] T cad47beffa4447a69d5d7cb1055ed3b3 P 86df19358ca742489cce32be42a6788e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3851b749dbbb4d548f57e50735ed8bca in term 1.
I20260928 14:53:18.717419 22359 raft_consensus.cc:2454] T cad47beffa4447a69d5d7cb1055ed3b3 P e70bc98543884c059729cebbd0010cf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3851b749dbbb4d548f57e50735ed8bca in term 1.
I20260928 14:53:18.717469 22432 leader_election.cc:302] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3851b749dbbb4d548f57e50735ed8bca, 86df19358ca742489cce32be42a6788e; no voters:
I20260928 14:53:18.717571 22572 raft_consensus.cc:2781] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:18.717759 22572 raft_consensus.cc:686] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca [term 1 LEADER]: Becoming Leader. State: Replica: 3851b749dbbb4d548f57e50735ed8bca, State: Running, Role: LEADER
I20260928 14:53:18.717890 22572 consensus_queue.cc:237] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } }
I20260928 14:53:18.718811 22061 catalog_manager.cc:5495] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca reported cstate change: term changed from 0 to 1, leader changed from <none> to 3851b749dbbb4d548f57e50735ed8bca (127.20.116.131). New cstate: current_term: 1 leader_uuid: "3851b749dbbb4d548f57e50735ed8bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } health_report { overall_health: UNKNOWN } } }
I20260928 14:53:18.728259 22565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.725819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47380 (local address 127.20.116.131:38191)
0928 14:53:18.725871 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:18.725873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:18.725888 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:53:18.725960 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:18.725961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:18.725976 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:18.726035 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:18.726176 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.726682 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.727554 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.727743 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.727793 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.727973 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:18.728063 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:18.728133 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:18.728169 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 14:53:18.728186 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 14:53:18.729935 22219 raft_consensus.cc:1261] T cad47beffa4447a69d5d7cb1055ed3b3 P 86df19358ca742489cce32be42a6788e [term 1 FOLLOWER]: Refusing update from remote peer 3851b749dbbb4d548f57e50735ed8bca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:18.729964 22359 raft_consensus.cc:1261] T cad47beffa4447a69d5d7cb1055ed3b3 P e70bc98543884c059729cebbd0010cf9 [term 1 FOLLOWER]: Refusing update from remote peer 3851b749dbbb4d548f57e50735ed8bca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:18.730271 22572 consensus_queue.cc:1035] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca [LEADER]: Connected to new peer: Peer: permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:18.730434 22578 consensus_queue.cc:1035] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca [LEADER]: Connected to new peer: Peer: permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:18.733669 22582 mvcc.cc:205] Tried to move back new op lower bound from 7334327085995372544 to 7334327085949128704. Current Snapshot: MvccSnapshot[applied={T|T < 7334327085995372544 or (T in {7334327085995372544})}]
I20260928 14:53:18.733796 22583 mvcc.cc:205] Tried to move back new op lower bound from 7334327085995372544 to 7334327085949128704. Current Snapshot: MvccSnapshot[applied={T|T < 7334327085995372544}]
I20260928 14:53:18.736231 22062 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:18.736334 22062 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:18.737169 22062 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec: Sending DeleteTablet for 3 replicas of tablet cad47beffa4447a69d5d7cb1055ed3b3
I20260928 14:53:18.737522 22199 tablet_service.cc:1514] Processing DeleteTablet for tablet cad47beffa4447a69d5d7cb1055ed3b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:18 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:32958
I20260928 14:53:18.737548 22339 tablet_service.cc:1514] Processing DeleteTablet for tablet cad47beffa4447a69d5d7cb1055ed3b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:18 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:33572
I20260928 14:53:18.737782 22592 tablet_replica.cc:330] T cad47beffa4447a69d5d7cb1055ed3b3 P e70bc98543884c059729cebbd0010cf9: stopping tablet replica
I20260928 14:53:18.737882 22592 raft_consensus.cc:2229] T cad47beffa4447a69d5d7cb1055ed3b3 P e70bc98543884c059729cebbd0010cf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:18.737859 22479 tablet_service.cc:1514] Processing DeleteTablet for tablet cad47beffa4447a69d5d7cb1055ed3b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:18 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:47366
I20260928 14:53:18.738018 22592 raft_consensus.cc:2258] T cad47beffa4447a69d5d7cb1055ed3b3 P e70bc98543884c059729cebbd0010cf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:18.738137 22593 tablet_replica.cc:330] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca: stopping tablet replica
I20260928 14:53:18.738269 22593 raft_consensus.cc:2229] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:53:18.738271 22062 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 14:53:18.738374 22593 raft_consensus.cc:2258] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 14:53:18.738404 22062 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:53:18.738986 22592 ts_tablet_manager.cc:1905] T cad47beffa4447a69d5d7cb1055ed3b3 P e70bc98543884c059729cebbd0010cf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:18.739708 22199 tablet_service.cc:1467] Processing CreateTablet for tablet 7a715986de464c8f8800d1e9aa140cd0 (DEFAULT_TABLE table=test-table [id=1be8309182c74aa29f487367e61a6e35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:18.739825 22199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a715986de464c8f8800d1e9aa140cd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:18.740739 22569 tablet_bootstrap.cc:492] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e: Bootstrap starting.
I20260928 14:53:18.741091 22339 tablet_service.cc:1467] Processing CreateTablet for tablet 7a715986de464c8f8800d1e9aa140cd0 (DEFAULT_TABLE table=test-table [id=1be8309182c74aa29f487367e61a6e35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:18.741178 22569 tablet_bootstrap.cc:654] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:18.741201 22339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a715986de464c8f8800d1e9aa140cd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:18.741210 22479 tablet_service.cc:1467] Processing CreateTablet for tablet 7a715986de464c8f8800d1e9aa140cd0 (DEFAULT_TABLE table=test-table [id=1be8309182c74aa29f487367e61a6e35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:18.741353 22479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a715986de464c8f8800d1e9aa140cd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:18.742449 22573 tablet_bootstrap.cc:492] T 7a715986de464c8f8800d1e9aa140cd0 P e70bc98543884c059729cebbd0010cf9: Bootstrap starting.
I20260928 14:53:18.742532 22568 tablet_bootstrap.cc:492] T 7a715986de464c8f8800d1e9aa140cd0 P 3851b749dbbb4d548f57e50735ed8bca: Bootstrap starting.
I20260928 14:53:18.742628 22591 tablet_replica.cc:330] T cad47beffa4447a69d5d7cb1055ed3b3 P 86df19358ca742489cce32be42a6788e: stopping tablet replica
I20260928 14:53:18.742735 22591 raft_consensus.cc:2229] T cad47beffa4447a69d5d7cb1055ed3b3 P 86df19358ca742489cce32be42a6788e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:18.742841 22591 raft_consensus.cc:2258] T cad47beffa4447a69d5d7cb1055ed3b3 P 86df19358ca742489cce32be42a6788e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:18.742849 22573 tablet_bootstrap.cc:654] T 7a715986de464c8f8800d1e9aa140cd0 P e70bc98543884c059729cebbd0010cf9: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:18.742964 22568 tablet_bootstrap.cc:654] T 7a715986de464c8f8800d1e9aa140cd0 P 3851b749dbbb4d548f57e50735ed8bca: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:18.743393 22591 ts_tablet_manager.cc:1905] T cad47beffa4447a69d5d7cb1055ed3b3 P 86df19358ca742489cce32be42a6788e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:18.743723 22593 ts_tablet_manager.cc:1905] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:18.743760 22592 ts_tablet_manager.cc:1918] T cad47beffa4447a69d5d7cb1055ed3b3 P e70bc98543884c059729cebbd0010cf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:53:18.743927 22592 log.cc:1196] T cad47beffa4447a69d5d7cb1055ed3b3 P e70bc98543884c059729cebbd0010cf9: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/cad47beffa4447a69d5d7cb1055ed3b3
I20260928 14:53:18.744257 22592 ts_tablet_manager.cc:1939] T cad47beffa4447a69d5d7cb1055ed3b3 P e70bc98543884c059729cebbd0010cf9: Deleting consensus metadata
I20260928 14:53:18.744727 22049 catalog_manager.cc:4841] TS e70bc98543884c059729cebbd0010cf9 (127.20.116.130:44025): tablet cad47beffa4447a69d5d7cb1055ed3b3 (table test-table [id=ad425fa2b6ce43f6acf7261a680fb82c]) successfully deleted
I20260928 14:53:18.744997 22591 ts_tablet_manager.cc:1918] T cad47beffa4447a69d5d7cb1055ed3b3 P 86df19358ca742489cce32be42a6788e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:53:18.745083 22591 log.cc:1196] T cad47beffa4447a69d5d7cb1055ed3b3 P 86df19358ca742489cce32be42a6788e: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-0/wal/wals/cad47beffa4447a69d5d7cb1055ed3b3
I20260928 14:53:18.745349 22573 tablet_bootstrap.cc:492] T 7a715986de464c8f8800d1e9aa140cd0 P e70bc98543884c059729cebbd0010cf9: No bootstrap required, opened a new log
I20260928 14:53:18.745378 22591 ts_tablet_manager.cc:1939] T cad47beffa4447a69d5d7cb1055ed3b3 P 86df19358ca742489cce32be42a6788e: Deleting consensus metadata
I20260928 14:53:18.745397 22573 ts_tablet_manager.cc:1397] T 7a715986de464c8f8800d1e9aa140cd0 P e70bc98543884c059729cebbd0010cf9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 14:53:18.745560 22593 ts_tablet_manager.cc:1918] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:53:18.745543 22573 raft_consensus.cc:348] T 7a715986de464c8f8800d1e9aa140cd0 P e70bc98543884c059729cebbd0010cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } }
I20260928 14:53:18.745608 22573 raft_consensus.cc:374] T 7a715986de464c8f8800d1e9aa140cd0 P e70bc98543884c059729cebbd0010cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:18.745616 22593 log.cc:1196] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/cad47beffa4447a69d5d7cb1055ed3b3
I20260928 14:53:18.745630 22573 raft_consensus.cc:729] T 7a715986de464c8f8800d1e9aa140cd0 P e70bc98543884c059729cebbd0010cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e70bc98543884c059729cebbd0010cf9, State: Initialized, Role: FOLLOWER
I20260928 14:53:18.745673 22573 consensus_queue.cc:260] T 7a715986de464c8f8800d1e9aa140cd0 P e70bc98543884c059729cebbd0010cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } }
I20260928 14:53:18.745750 22573 ts_tablet_manager.cc:1428] T 7a715986de464c8f8800d1e9aa140cd0 P e70bc98543884c059729cebbd0010cf9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:18.745754 22048 catalog_manager.cc:4841] TS 86df19358ca742489cce32be42a6788e (127.20.116.129:41677): tablet cad47beffa4447a69d5d7cb1055ed3b3 (table test-table [id=ad425fa2b6ce43f6acf7261a680fb82c]) successfully deleted
I20260928 14:53:18.745968 22593 ts_tablet_manager.cc:1939] T cad47beffa4447a69d5d7cb1055ed3b3 P 3851b749dbbb4d548f57e50735ed8bca: Deleting consensus metadata
I20260928 14:53:18.746312 22047 catalog_manager.cc:4841] TS 3851b749dbbb4d548f57e50735ed8bca (127.20.116.131:38191): tablet cad47beffa4447a69d5d7cb1055ed3b3 (table test-table [id=ad425fa2b6ce43f6acf7261a680fb82c]) successfully deleted
I20260928 14:53:18.746562 22568 tablet_bootstrap.cc:492] T 7a715986de464c8f8800d1e9aa140cd0 P 3851b749dbbb4d548f57e50735ed8bca: No bootstrap required, opened a new log
I20260928 14:53:18.746613 22568 ts_tablet_manager.cc:1397] T 7a715986de464c8f8800d1e9aa140cd0 P 3851b749dbbb4d548f57e50735ed8bca: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 14:53:18.746729 22569 tablet_bootstrap.cc:492] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e: No bootstrap required, opened a new log
I20260928 14:53:18.746771 22569 ts_tablet_manager.cc:1397] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 14:53:18.746773 22568 raft_consensus.cc:348] T 7a715986de464c8f8800d1e9aa140cd0 P 3851b749dbbb4d548f57e50735ed8bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } }
I20260928 14:53:18.746841 22568 raft_consensus.cc:374] T 7a715986de464c8f8800d1e9aa140cd0 P 3851b749dbbb4d548f57e50735ed8bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:18.746867 22568 raft_consensus.cc:729] T 7a715986de464c8f8800d1e9aa140cd0 P 3851b749dbbb4d548f57e50735ed8bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3851b749dbbb4d548f57e50735ed8bca, State: Initialized, Role: FOLLOWER
I20260928 14:53:18.746917 22568 consensus_queue.cc:260] T 7a715986de464c8f8800d1e9aa140cd0 P 3851b749dbbb4d548f57e50735ed8bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } }
I20260928 14:53:18.747006 22568 ts_tablet_manager.cc:1428] T 7a715986de464c8f8800d1e9aa140cd0 P 3851b749dbbb4d548f57e50735ed8bca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:18.747133 22569 raft_consensus.cc:348] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } }
I20260928 14:53:18.747205 22569 raft_consensus.cc:374] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:18.747226 22569 raft_consensus.cc:729] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86df19358ca742489cce32be42a6788e, State: Initialized, Role: FOLLOWER
I20260928 14:53:18.747272 22569 consensus_queue.cc:260] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } }
I20260928 14:53:18.747382 22569 ts_tablet_manager.cc:1428] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260928 14:53:18.818401 22546 tablet.cc:2369] T 7a715986de464c8f8800d1e9aa140cd0 P 3851b749dbbb4d548f57e50735ed8bca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:53:18.873984 22406 tablet.cc:2369] T 7a715986de464c8f8800d1e9aa140cd0 P e70bc98543884c059729cebbd0010cf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:53:18.897217 22574 raft_consensus.cc:482] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:53:18.897315 22574 raft_consensus.cc:504] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } }
I20260928 14:53:18.897732 22574 leader_election.cc:288] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e70bc98543884c059729cebbd0010cf9 (127.20.116.130:44025), 3851b749dbbb4d548f57e50735ed8bca (127.20.116.131:38191)
I20260928 14:53:18.900833 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.898001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:60195 (local address 127.20.116.130:44025)
0928 14:53:18.898051 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:18.898054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:18.898201 (+ 147us) server_negotiation.cc:407] Connection header received
0928 14:53:18.898323 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:18.898324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:18.898334 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:18.898409 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:18.898648 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.899120 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.900100 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.900463 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.900553 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.900563 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:18.900720 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 14:53:18.900760 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:53:18.900872 22565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.898016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:36033 (local address 127.20.116.131:38191)
0928 14:53:18.898077 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:18.898079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:18.898207 (+ 128us) server_negotiation.cc:407] Connection header received
0928 14:53:18.898293 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:18.898294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:18.898314 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:18.898376 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:18.898561 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.899038 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.900110 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.900477 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.900573 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.900583 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:18.900760 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 14:53:18.900802 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 14:53:18.900910 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.897917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:44025 (local address 127.20.116.129:60195)
0928 14:53:18.898098 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:18.898110 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:18.898236 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:18.898413 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:18.898414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:18.898416 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:18.898497 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.898499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.899222 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.899223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.900005 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.900007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.900553 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.900554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.900632 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.900636 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:53:18.900711 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 14:53:18.900753 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":1,"thread_start_us":58,"threads_started":1}
I20260928 14:53:18.900909 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.897960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:38191 (local address 127.20.116.129:36033)
0928 14:53:18.898103 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:18.898112 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:18.898223 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:18.898382 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:18.898384 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:18.898386 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:18.898465 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.898467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.899143 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.899144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.900007 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.900009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.900599 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.900600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.900672 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.900675 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:53:18.900750 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 14:53:18.900790 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":3,"thread_start_us":72,"threads_started":1}
I20260928 14:53:18.901214 22359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a715986de464c8f8800d1e9aa140cd0" candidate_uuid: "86df19358ca742489cce32be42a6788e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e70bc98543884c059729cebbd0010cf9" is_pre_election: true
I20260928 14:53:18.901321 22359 raft_consensus.cc:2454] T 7a715986de464c8f8800d1e9aa140cd0 P e70bc98543884c059729cebbd0010cf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86df19358ca742489cce32be42a6788e in term 0.
I20260928 14:53:18.901520 22153 leader_election.cc:302] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86df19358ca742489cce32be42a6788e, e70bc98543884c059729cebbd0010cf9; no voters:
I20260928 14:53:18.901486 22499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a715986de464c8f8800d1e9aa140cd0" candidate_uuid: "86df19358ca742489cce32be42a6788e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3851b749dbbb4d548f57e50735ed8bca" is_pre_election: true
I20260928 14:53:18.901648 22499 raft_consensus.cc:2454] T 7a715986de464c8f8800d1e9aa140cd0 P 3851b749dbbb4d548f57e50735ed8bca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86df19358ca742489cce32be42a6788e in term 0.
I20260928 14:53:18.901664 22574 raft_consensus.cc:2781] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:53:18.901700 22574 raft_consensus.cc:482] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:53:18.901732 22574 raft_consensus.cc:3037] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:18.902428 22574 raft_consensus.cc:504] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } }
I20260928 14:53:18.902616 22574 leader_election.cc:288] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e [CANDIDATE]: Term 1 election: Requested vote from peers e70bc98543884c059729cebbd0010cf9 (127.20.116.130:44025), 3851b749dbbb4d548f57e50735ed8bca (127.20.116.131:38191)
I20260928 14:53:18.902823 22499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a715986de464c8f8800d1e9aa140cd0" candidate_uuid: "86df19358ca742489cce32be42a6788e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3851b749dbbb4d548f57e50735ed8bca"
I20260928 14:53:18.902842 22359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a715986de464c8f8800d1e9aa140cd0" candidate_uuid: "86df19358ca742489cce32be42a6788e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e70bc98543884c059729cebbd0010cf9"
I20260928 14:53:18.902911 22499 raft_consensus.cc:3037] T 7a715986de464c8f8800d1e9aa140cd0 P 3851b749dbbb4d548f57e50735ed8bca [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:18.902911 22359 raft_consensus.cc:3037] T 7a715986de464c8f8800d1e9aa140cd0 P e70bc98543884c059729cebbd0010cf9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:18.903689 22499 raft_consensus.cc:2454] T 7a715986de464c8f8800d1e9aa140cd0 P 3851b749dbbb4d548f57e50735ed8bca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86df19358ca742489cce32be42a6788e in term 1.
I20260928 14:53:18.903687 22359 raft_consensus.cc:2454] T 7a715986de464c8f8800d1e9aa140cd0 P e70bc98543884c059729cebbd0010cf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86df19358ca742489cce32be42a6788e in term 1.
I20260928 14:53:18.903868 22151 leader_election.cc:302] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3851b749dbbb4d548f57e50735ed8bca, 86df19358ca742489cce32be42a6788e; no voters:
I20260928 14:53:18.903996 22574 raft_consensus.cc:2781] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:18.904152 22574 raft_consensus.cc:686] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e [term 1 LEADER]: Becoming Leader. State: Replica: 86df19358ca742489cce32be42a6788e, State: Running, Role: LEADER
I20260928 14:53:18.904208 22574 consensus_queue.cc:237] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } }
I20260928 14:53:18.904980 22061 catalog_manager.cc:5495] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e reported cstate change: term changed from 0 to 1, leader changed from <none> to 86df19358ca742489cce32be42a6788e (127.20.116.129). New cstate: current_term: 1 leader_uuid: "86df19358ca742489cce32be42a6788e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } health_report { overall_health: UNKNOWN } } }
Sep 28 14:53:18 dist-test-slave-pxl6 krb5kdc[22017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607197, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:18.947214 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.939510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43406 (local address 127.20.116.190:41203)
0928 14:53:18.939773 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:18.939775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:18.939846 (+ 71us) server_negotiation.cc:407] Connection header received
0928 14:53:18.939954 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:18.939955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:18.939967 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:18.940054 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:18.940623 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.941160 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.941895 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.942097 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.942185 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.944519 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:18.944521 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:18.944521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:18.944529 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:18.945868 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:18.946303 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:18.946304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:18.946305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:18.946363 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:18.946605 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:18.946606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:18.946606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:18.946721 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:18.946800 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:18.946976 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 14:53:18.947101 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":240}
I20260928 14:53:18.947283 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.939420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:41203 (local address 127.0.0.1:43406)
0928 14:53:18.939697 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:18.939760 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:18.939869 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:18.940062 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:18.940063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:18.940436 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:18.940503 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.940506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.941255 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.941256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.941807 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.941808 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.942187 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.942188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.942247 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.942933 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:18.942938 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:18.944416 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:18.945972 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:18.945982 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:18.945985 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:18.946211 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:18.946430 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:18.946430 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:18.946431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:18.946493 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:18.946791 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:18.946800 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:18.946848 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 14:53:18.946950 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 14:53:18.947158 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":98,"thread_start_us":131,"threads_started":1}
I20260928 14:53:18.949394 22061 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:43406:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 14:53:18.949616 22061 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:53:18.951186 22199 tablet_service.cc:1467] Processing CreateTablet for tablet d3e1276e258e4ace8f2fd7fbfd2db3ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f3ea2d0cfcb48a8aa7f718574a621fa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:53:18.951200 22479 tablet_service.cc:1467] Processing CreateTablet for tablet d3e1276e258e4ace8f2fd7fbfd2db3ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f3ea2d0cfcb48a8aa7f718574a621fa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:53:18.951355 22479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3e1276e258e4ace8f2fd7fbfd2db3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:18.951355 22199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3e1276e258e4ace8f2fd7fbfd2db3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:18.952903 22569 tablet_bootstrap.cc:492] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 86df19358ca742489cce32be42a6788e: Bootstrap starting.
I20260928 14:53:18.952991 22339 tablet_service.cc:1467] Processing CreateTablet for tablet d3e1276e258e4ace8f2fd7fbfd2db3ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f3ea2d0cfcb48a8aa7f718574a621fa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:53:18.953148 22339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3e1276e258e4ace8f2fd7fbfd2db3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:18.953336 22569 tablet_bootstrap.cc:654] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 86df19358ca742489cce32be42a6788e: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:18.953707 22568 tablet_bootstrap.cc:492] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca: Bootstrap starting.
I20260928 14:53:18.954061 22569 tablet_bootstrap.cc:492] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 86df19358ca742489cce32be42a6788e: No bootstrap required, opened a new log
I20260928 14:53:18.954058 22568 tablet_bootstrap.cc:654] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:18.954108 22569 ts_tablet_manager.cc:1397] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 86df19358ca742489cce32be42a6788e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:18.954282 22569 raft_consensus.cc:348] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 86df19358ca742489cce32be42a6788e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } }
I20260928 14:53:18.954356 22569 raft_consensus.cc:374] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 86df19358ca742489cce32be42a6788e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:18.954378 22569 raft_consensus.cc:729] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 86df19358ca742489cce32be42a6788e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86df19358ca742489cce32be42a6788e, State: Initialized, Role: FOLLOWER
I20260928 14:53:18.954367 22573 tablet_bootstrap.cc:492] T d3e1276e258e4ace8f2fd7fbfd2db3ad P e70bc98543884c059729cebbd0010cf9: Bootstrap starting.
I20260928 14:53:18.954421 22569 consensus_queue.cc:260] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 86df19358ca742489cce32be42a6788e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } }
I20260928 14:53:18.954555 22569 ts_tablet_manager.cc:1428] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 86df19358ca742489cce32be42a6788e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:18.954615 22574 tablet_replica.cc:439] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 86df19358ca742489cce32be42a6788e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } } }
I20260928 14:53:18.954700 22574 tablet_replica.cc:442] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 86df19358ca742489cce32be42a6788e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:53:18.954782 22573 tablet_bootstrap.cc:654] T d3e1276e258e4ace8f2fd7fbfd2db3ad P e70bc98543884c059729cebbd0010cf9: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:18.954958 22568 tablet_bootstrap.cc:492] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca: No bootstrap required, opened a new log
I20260928 14:53:18.955013 22568 ts_tablet_manager.cc:1397] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 14:53:18.955147 22568 raft_consensus.cc:348] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } }
I20260928 14:53:18.955243 22568 raft_consensus.cc:374] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:18.955262 22568 raft_consensus.cc:729] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3851b749dbbb4d548f57e50735ed8bca, State: Initialized, Role: FOLLOWER
I20260928 14:53:18.955299 22568 consensus_queue.cc:260] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } }
I20260928 14:53:18.955392 22568 ts_tablet_manager.cc:1428] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:18.955556 22573 tablet_bootstrap.cc:492] T d3e1276e258e4ace8f2fd7fbfd2db3ad P e70bc98543884c059729cebbd0010cf9: No bootstrap required, opened a new log
I20260928 14:53:18.955510 22586 tablet_replica.cc:439] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } } }
I20260928 14:53:18.955600 22586 tablet_replica.cc:442] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:53:18.955605 22573 ts_tablet_manager.cc:1397] T d3e1276e258e4ace8f2fd7fbfd2db3ad P e70bc98543884c059729cebbd0010cf9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 14:53:18.955752 22573 raft_consensus.cc:348] T d3e1276e258e4ace8f2fd7fbfd2db3ad P e70bc98543884c059729cebbd0010cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } }
I20260928 14:53:18.955834 22573 raft_consensus.cc:374] T d3e1276e258e4ace8f2fd7fbfd2db3ad P e70bc98543884c059729cebbd0010cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:18.955861 22573 raft_consensus.cc:729] T d3e1276e258e4ace8f2fd7fbfd2db3ad P e70bc98543884c059729cebbd0010cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e70bc98543884c059729cebbd0010cf9, State: Initialized, Role: FOLLOWER
I20260928 14:53:18.955901 22573 consensus_queue.cc:260] T d3e1276e258e4ace8f2fd7fbfd2db3ad P e70bc98543884c059729cebbd0010cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } }
I20260928 14:53:18.955994 22573 ts_tablet_manager.cc:1428] T d3e1276e258e4ace8f2fd7fbfd2db3ad P e70bc98543884c059729cebbd0010cf9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:18.956059 22576 tablet_replica.cc:439] T d3e1276e258e4ace8f2fd7fbfd2db3ad P e70bc98543884c059729cebbd0010cf9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } } }
I20260928 14:53:18.956161 22576 tablet_replica.cc:442] T d3e1276e258e4ace8f2fd7fbfd2db3ad P e70bc98543884c059729cebbd0010cf9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:53:18.971709 22586 raft_consensus.cc:482] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:53:18.971786 22586 raft_consensus.cc:504] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } }
I20260928 14:53:18.971990 22586 leader_election.cc:288] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86df19358ca742489cce32be42a6788e (127.20.116.129:41677), e70bc98543884c059729cebbd0010cf9 (127.20.116.130:44025)
I20260928 14:53:18.972182 22219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e1276e258e4ace8f2fd7fbfd2db3ad" candidate_uuid: "3851b749dbbb4d548f57e50735ed8bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86df19358ca742489cce32be42a6788e" is_pre_election: true
I20260928 14:53:18.972224 22359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e1276e258e4ace8f2fd7fbfd2db3ad" candidate_uuid: "3851b749dbbb4d548f57e50735ed8bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e70bc98543884c059729cebbd0010cf9" is_pre_election: true
I20260928 14:53:18.972296 22219 raft_consensus.cc:2454] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 86df19358ca742489cce32be42a6788e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3851b749dbbb4d548f57e50735ed8bca in term 0.
I20260928 14:53:18.972294 22359 raft_consensus.cc:2454] T d3e1276e258e4ace8f2fd7fbfd2db3ad P e70bc98543884c059729cebbd0010cf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3851b749dbbb4d548f57e50735ed8bca in term 0.
I20260928 14:53:18.972448 22432 leader_election.cc:302] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3851b749dbbb4d548f57e50735ed8bca, 86df19358ca742489cce32be42a6788e; no voters:
I20260928 14:53:18.972541 22586 raft_consensus.cc:2781] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:53:18.972573 22586 raft_consensus.cc:482] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:53:18.972601 22586 raft_consensus.cc:3037] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:18.973284 22586 raft_consensus.cc:504] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } }
I20260928 14:53:18.973434 22586 leader_election.cc:288] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca [CANDIDATE]: Term 1 election: Requested vote from peers 86df19358ca742489cce32be42a6788e (127.20.116.129:41677), e70bc98543884c059729cebbd0010cf9 (127.20.116.130:44025)
I20260928 14:53:18.973614 22219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e1276e258e4ace8f2fd7fbfd2db3ad" candidate_uuid: "3851b749dbbb4d548f57e50735ed8bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86df19358ca742489cce32be42a6788e"
I20260928 14:53:18.973630 22359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e1276e258e4ace8f2fd7fbfd2db3ad" candidate_uuid: "3851b749dbbb4d548f57e50735ed8bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e70bc98543884c059729cebbd0010cf9"
I20260928 14:53:18.973693 22219 raft_consensus.cc:3037] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 86df19358ca742489cce32be42a6788e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:18.973706 22359 raft_consensus.cc:3037] T d3e1276e258e4ace8f2fd7fbfd2db3ad P e70bc98543884c059729cebbd0010cf9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:18.974298 22359 raft_consensus.cc:2454] T d3e1276e258e4ace8f2fd7fbfd2db3ad P e70bc98543884c059729cebbd0010cf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3851b749dbbb4d548f57e50735ed8bca in term 1.
I20260928 14:53:18.974378 22219 raft_consensus.cc:2454] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 86df19358ca742489cce32be42a6788e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3851b749dbbb4d548f57e50735ed8bca in term 1.
I20260928 14:53:18.974478 22433 leader_election.cc:302] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3851b749dbbb4d548f57e50735ed8bca, e70bc98543884c059729cebbd0010cf9; no voters:
I20260928 14:53:18.974611 22586 raft_consensus.cc:2781] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:18.974668 22586 raft_consensus.cc:686] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca [term 1 LEADER]: Becoming Leader. State: Replica: 3851b749dbbb4d548f57e50735ed8bca, State: Running, Role: LEADER
I20260928 14:53:18.974725 22586 consensus_queue.cc:237] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } }
I20260928 14:53:18.975080 22578 tablet_replica.cc:439] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca: TxnStatusTablet state changed. Reason: New leader 3851b749dbbb4d548f57e50735ed8bca. Latest consensus state: current_term: 1 leader_uuid: "3851b749dbbb4d548f57e50735ed8bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } } }
I20260928 14:53:18.975183 22578 tablet_replica.cc:442] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:53:18.975386 22611 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 14:53:18.975353 22062 catalog_manager.cc:5495] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca reported cstate change: term changed from 0 to 1, leader changed from <none> to 3851b749dbbb4d548f57e50735ed8bca (127.20.116.131). New cstate: current_term: 1 leader_uuid: "3851b749dbbb4d548f57e50735ed8bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } health_report { overall_health: HEALTHY } } }
I20260928 14:53:18.975521 22611 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 14:53:18.985183 22565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.982498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47394 (local address 127.20.116.131:38191)
0928 14:53:18.982577 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:18.982579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:18.982631 (+ 52us) server_negotiation.cc:407] Connection header received
0928 14:53:18.982675 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:18.982676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:18.982684 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:18.982734 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:18.982921 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.983392 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.984482 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.984672 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.984848 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.984927 (+ 79us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:18.984961 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:18.985008 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:18.985103 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 14:53:18.985124 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:53:18.985219 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.982438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:38191 (local address 127.0.0.1:47394)
0928 14:53:18.982543 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:18.982555 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:18.982608 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:18.982757 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:18.982758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:18.982759 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:53:18.982814 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.982816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.983471 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.983472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.984398 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.984400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.984753 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.984754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.984826 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.984845 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:53:18.985029 (+ 184us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:53:18.985037 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 14:53:18.985088 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 14:53:18.985137 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 14:53:18.986289 22219 raft_consensus.cc:1261] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 86df19358ca742489cce32be42a6788e [term 1 FOLLOWER]: Refusing update from remote peer 3851b749dbbb4d548f57e50735ed8bca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:18.986325 22359 raft_consensus.cc:1261] T d3e1276e258e4ace8f2fd7fbfd2db3ad P e70bc98543884c059729cebbd0010cf9 [term 1 FOLLOWER]: Refusing update from remote peer 3851b749dbbb4d548f57e50735ed8bca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:18.986552 22578 consensus_queue.cc:1035] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca [LEADER]: Connected to new peer: Peer: permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:18.986711 22586 consensus_queue.cc:1035] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca [LEADER]: Connected to new peer: Peer: permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:18.987222 22574 tablet_replica.cc:439] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 86df19358ca742489cce32be42a6788e: TxnStatusTablet state changed. Reason: New leader 3851b749dbbb4d548f57e50735ed8bca. Latest consensus state: current_term: 1 leader_uuid: "3851b749dbbb4d548f57e50735ed8bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } } }
I20260928 14:53:18.987310 22574 tablet_replica.cc:442] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 86df19358ca742489cce32be42a6788e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:53:18.987389 22576 tablet_replica.cc:439] T d3e1276e258e4ace8f2fd7fbfd2db3ad P e70bc98543884c059729cebbd0010cf9: TxnStatusTablet state changed. Reason: New leader 3851b749dbbb4d548f57e50735ed8bca. Latest consensus state: current_term: 1 leader_uuid: "3851b749dbbb4d548f57e50735ed8bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } } }
I20260928 14:53:18.987488 22576 tablet_replica.cc:442] T d3e1276e258e4ace8f2fd7fbfd2db3ad P e70bc98543884c059729cebbd0010cf9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:53:18.987950 22586 tablet_replica.cc:439] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3851b749dbbb4d548f57e50735ed8bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } } }
I20260928 14:53:18.988039 22586 tablet_replica.cc:442] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:53:18.988245 22578 tablet_replica.cc:439] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3851b749dbbb4d548f57e50735ed8bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } } }
I20260928 14:53:18.988335 22578 tablet_replica.cc:442] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 3851b749dbbb4d548f57e50735ed8bca: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:53:18.988451 22581 mvcc.cc:205] Tried to move back new op lower bound from 7334327087045836800 to 7334327087000903680. Current Snapshot: MvccSnapshot[applied={T|T < 7334327087045836800}]
I20260928 14:53:18.988662 22576 tablet_replica.cc:439] T d3e1276e258e4ace8f2fd7fbfd2db3ad P e70bc98543884c059729cebbd0010cf9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3851b749dbbb4d548f57e50735ed8bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } } }
I20260928 14:53:18.988744 22576 tablet_replica.cc:442] T d3e1276e258e4ace8f2fd7fbfd2db3ad P e70bc98543884c059729cebbd0010cf9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:53:18.989208 22574 tablet_replica.cc:439] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 86df19358ca742489cce32be42a6788e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3851b749dbbb4d548f57e50735ed8bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86df19358ca742489cce32be42a6788e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41677 } } peers { permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 } } peers { permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 } } }
I20260928 14:53:18.989292 22574 tablet_replica.cc:442] T d3e1276e258e4ace8f2fd7fbfd2db3ad P 86df19358ca742489cce32be42a6788e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:53:18.993028 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.990488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32970 (local address 127.20.116.129:41677)
0928 14:53:18.990565 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:18.990567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:18.990575 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:53:18.990618 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:18.990619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:18.990646 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:18.990727 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:18.990832 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.991305 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.992182 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.992399 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.992499 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.992669 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:18.992787 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:18.992869 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:18.992926 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 14:53:18.992945 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 14:53:18.998224 22565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.995738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:59273 (local address 127.20.116.131:38191)
0928 14:53:18.995792 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:18.995794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:18.995915 (+ 121us) server_negotiation.cc:407] Connection header received
0928 14:53:18.996006 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:18.996007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:18.996019 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:18.996070 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:18.996244 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.996709 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.997474 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.997681 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.997767 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.997950 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:18.997989 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:18.998047 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:18.998125 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 14:53:18.998149 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:53:18.998246 22618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:18.995656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:38191 (local address 127.20.116.129:59273)
0928 14:53:18.995848 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:18.995857 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:18.995929 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:18.996073 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:18.996074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:18.996075 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:53:18.996161 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.996162 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.996791 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.996792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.997386 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:18.997388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:18.997770 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:18.997771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:18.997837 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:18.997851 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:53:18.998062 (+ 211us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:53:18.998064 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:53:18.998119 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 14:53:18.998165 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":78,"threads_started":1}
I20260928 14:53:19.000973 22359 raft_consensus.cc:1261] T 7a715986de464c8f8800d1e9aa140cd0 P e70bc98543884c059729cebbd0010cf9 [term 1 FOLLOWER]: Refusing update from remote peer 86df19358ca742489cce32be42a6788e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:19.001065 22499 raft_consensus.cc:1261] T 7a715986de464c8f8800d1e9aa140cd0 P 3851b749dbbb4d548f57e50735ed8bca [term 1 FOLLOWER]: Refusing update from remote peer 86df19358ca742489cce32be42a6788e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:19.001260 22599 consensus_queue.cc:1035] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e70bc98543884c059729cebbd0010cf9" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:19.001370 22574 consensus_queue.cc:1035] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3851b749dbbb4d548f57e50735ed8bca" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:19.012022 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:19.009005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:57213 (local address 127.20.116.129:41677)
0928 14:53:19.009046 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:19.009048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:19.009160 (+ 112us) server_negotiation.cc:407] Connection header received
0928 14:53:19.009194 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:19.009194 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:19.009205 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:19.009273 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:19.009424 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:19.009951 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:19.011119 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:19.011472 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:19.011544 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:19.011766 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:19.011804 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:19.011859 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:19.011930 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 14:53:19.011949 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:53:19.012082 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:19.008941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:41677 (local address 127.20.116.131:57213)
0928 14:53:19.009033 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:19.009046 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:19.009101 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:19.009277 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:19.009278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:19.009279 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:53:19.009341 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:19.009343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:19.010058 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:19.010059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:19.011032 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:19.011036 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:19.011561 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:19.011562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:19.011649 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:19.011666 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:53:19.011876 (+ 210us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:53:19.011878 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:53:19.011929 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 14:53:19.011997 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 14:53:19.022154 22062 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:19.022280 22062 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:19.022936 22062 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50c60f6555694292b065664094fae6ec: Sending DeleteTablet for 3 replicas of tablet 7a715986de464c8f8800d1e9aa140cd0
I20260928 14:53:19.023228 22479 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a715986de464c8f8800d1e9aa140cd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:19 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:47366
I20260928 14:53:19.023288 22199 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a715986de464c8f8800d1e9aa140cd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:19 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:32958
I20260928 14:53:19.023381 22591 tablet_replica.cc:330] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e: stopping tablet replica
I20260928 14:53:19.023427 22591 raft_consensus.cc:2229] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:53:19.023512 22591 raft_consensus.cc:2258] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:19.023648 22339 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a715986de464c8f8800d1e9aa140cd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:19 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:33572
I20260928 14:53:19.023730 22592 tablet_replica.cc:330] T 7a715986de464c8f8800d1e9aa140cd0 P e70bc98543884c059729cebbd0010cf9: stopping tablet replica
I20260928 14:53:19.023770 22592 raft_consensus.cc:2229] T 7a715986de464c8f8800d1e9aa140cd0 P e70bc98543884c059729cebbd0010cf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:19.023803 22592 raft_consensus.cc:2258] T 7a715986de464c8f8800d1e9aa140cd0 P e70bc98543884c059729cebbd0010cf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:19.023815 22593 tablet_replica.cc:330] T 7a715986de464c8f8800d1e9aa140cd0 P 3851b749dbbb4d548f57e50735ed8bca: stopping tablet replica
I20260928 14:53:19.023880 22593 raft_consensus.cc:2229] T 7a715986de464c8f8800d1e9aa140cd0 P 3851b749dbbb4d548f57e50735ed8bca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:19.023916 22593 raft_consensus.cc:2258] T 7a715986de464c8f8800d1e9aa140cd0 P 3851b749dbbb4d548f57e50735ed8bca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:19.024148 22592 ts_tablet_manager.cc:1905] T 7a715986de464c8f8800d1e9aa140cd0 P e70bc98543884c059729cebbd0010cf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:19.024704 22593 ts_tablet_manager.cc:1905] T 7a715986de464c8f8800d1e9aa140cd0 P 3851b749dbbb4d548f57e50735ed8bca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:19.025807 22592 ts_tablet_manager.cc:1918] T 7a715986de464c8f8800d1e9aa140cd0 P e70bc98543884c059729cebbd0010cf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:53:19.025861 22592 log.cc:1196] T 7a715986de464c8f8800d1e9aa140cd0 P e70bc98543884c059729cebbd0010cf9: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/7a715986de464c8f8800d1e9aa140cd0
I20260928 14:53:19.026075 22592 ts_tablet_manager.cc:1939] T 7a715986de464c8f8800d1e9aa140cd0 P e70bc98543884c059729cebbd0010cf9: Deleting consensus metadata
I20260928 14:53:19.026384 22593 ts_tablet_manager.cc:1918] T 7a715986de464c8f8800d1e9aa140cd0 P 3851b749dbbb4d548f57e50735ed8bca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:53:19.026433 22593 log.cc:1196] T 7a715986de464c8f8800d1e9aa140cd0 P 3851b749dbbb4d548f57e50735ed8bca: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/7a715986de464c8f8800d1e9aa140cd0
I20260928 14:53:19.026562 22049 catalog_manager.cc:4841] TS e70bc98543884c059729cebbd0010cf9 (127.20.116.130:44025): tablet 7a715986de464c8f8800d1e9aa140cd0 (table test-table [id=1be8309182c74aa29f487367e61a6e35]) successfully deleted
I20260928 14:53:19.026675 22593 ts_tablet_manager.cc:1939] T 7a715986de464c8f8800d1e9aa140cd0 P 3851b749dbbb4d548f57e50735ed8bca: Deleting consensus metadata
I20260928 14:53:19.027029 22047 catalog_manager.cc:4841] TS 3851b749dbbb4d548f57e50735ed8bca (127.20.116.131:38191): tablet 7a715986de464c8f8800d1e9aa140cd0 (table test-table [id=1be8309182c74aa29f487367e61a6e35]) successfully deleted
I20260928 14:53:19.027261 22591 ts_tablet_manager.cc:1905] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:19.028801 22591 ts_tablet_manager.cc:1918] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:53:19.028855 22591 log.cc:1196] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607192794321-20946-0/minicluster-data/ts-0/wal/wals/7a715986de464c8f8800d1e9aa140cd0
I20260928 14:53:19.029057 22591 ts_tablet_manager.cc:1939] T 7a715986de464c8f8800d1e9aa140cd0 P 86df19358ca742489cce32be42a6788e: Deleting consensus metadata
I20260928 14:53:19.029537 22048 catalog_manager.cc:4841] TS 86df19358ca742489cce32be42a6788e (127.20.116.129:41677): tablet 7a715986de464c8f8800d1e9aa140cd0 (table test-table [id=1be8309182c74aa29f487367e61a6e35]) successfully deleted
Sep 28 14:53:19 dist-test-slave-pxl6 krb5kdc[22017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607198, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.116.129@KRBTEST.COM
I20260928 14:53:19.040096 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:19.029611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32976 (local address 127.20.116.129:41677)
0928 14:53:19.029667 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:19.029669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:19.029982 (+ 313us) server_negotiation.cc:407] Connection header received
0928 14:53:19.030032 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:19.030033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:19.030045 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:19.030107 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:19.030834 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:19.031577 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:19.032660 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:19.032959 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:19.033035 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:19.036640 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:19.036651 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:19.036651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:19.036660 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:19.038737 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:19.039144 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:19.039148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:19.039149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:19.039217 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:19.039439 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:19.039440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:19.039441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:19.039639 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:19.039801 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:19.039882 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 14:53:19.039972 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 14:53:19.040355 22259 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32976
I20260928 14:53:19.049326 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:19.041374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43414 (local address 127.20.116.190:41203)
0928 14:53:19.041427 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:19.041430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:19.041643 (+ 213us) server_negotiation.cc:407] Connection header received
0928 14:53:19.041710 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:19.041711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:19.041723 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:19.041805 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:19.042477 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:19.043370 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:19.044443 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:19.044767 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:19.044840 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:19.046279 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:19.046280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:19.046281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:19.046289 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:19.048058 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:19.048462 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:19.048463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:19.048463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:19.048529 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:19.048752 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:19.048753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:19.048754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:19.048899 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:19.048997 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:19.049131 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 14:53:19.049216 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 14:53:19.049685 22062 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43414
I20260928 14:53:19.050495 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 22134
I20260928 14:53:19.056512 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 22274
I20260928 14:53:19.061800 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 22414
I20260928 14:53:19.067164 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 22030
2026-09-28T14:53:19Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2363 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:53:19 dist-test-slave-pxl6 krb5kdc[22631](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:53:19 dist-test-slave-pxl6 krb5kdc[22631](info): set up 2 sockets
Sep 28 14:53:19 dist-test-slave-pxl6 krb5kdc[22631](info): commencing operation
krb5kdc: starting...
W20260928 14:53:20.103206 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:53:20 dist-test-slave-pxl6 krb5kdc[22631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:53:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:53:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:53:20.224897 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:39027
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:39377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:39027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:20.306286 22644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:20.306501 22644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:20.306523 22644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:20.307667 22644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:53:20.307699 22644 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 14:53:20.307713 22644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:20.307725 22644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:20.307739 22644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:53:20.307750 22644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:53:20.309154 22644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:39377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:39027
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:39027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:20.309396 22644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:20.309629 22644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:20.314723 22649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:20.314769 22650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:20.314814 22652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:20.315003 22644 server_base.cc:1034] running on GCE node
I20260928 14:53:20.315230 22644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:20.315482 22644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:20.316660 22644 hybrid_clock.cc:648] HybridClock initialized: now 1790607200316643 us; error 35 us; skew 500 ppm
Sep 28 14:53:20 dist-test-slave-pxl6 krb5kdc[22631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607200, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:20.319679 22644 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:53:20.320144 22644 webserver.cc:460] Webserver started at http://127.20.116.190:38411/ using document root <none> and password file <none>
I20260928 14:53:20.320391 22644 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:20.320461 22644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:20.320600 22644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:20.321642 22644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "719aa42e8b754f3d8de1fd8b77b052f7"
format_stamp: "Formatted at 2026-09-28 14:53:20 on dist-test-slave-pxl6"
server_key: "0492fb4a76180da0af7ec9c0aaf2118b"
server_key_iv: "eb53af624fec6d6d6c61ba8e2810e647"
server_key_version: "encryptionkey@0"
I20260928 14:53:20.322001 22644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "719aa42e8b754f3d8de1fd8b77b052f7"
format_stamp: "Formatted at 2026-09-28 14:53:20 on dist-test-slave-pxl6"
server_key: "0492fb4a76180da0af7ec9c0aaf2118b"
server_key_iv: "eb53af624fec6d6d6c61ba8e2810e647"
server_key_version: "encryptionkey@0"
I20260928 14:53:20.323408 22644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:53:20.324209 22659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:20.324390 22644 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:20.324498 22644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "719aa42e8b754f3d8de1fd8b77b052f7"
format_stamp: "Formatted at 2026-09-28 14:53:20 on dist-test-slave-pxl6"
server_key: "0492fb4a76180da0af7ec9c0aaf2118b"
server_key_iv: "eb53af624fec6d6d6c61ba8e2810e647"
server_key_version: "encryptionkey@0"
I20260928 14:53:20.324572 22644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:20.335315 22644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:20.337808 22644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:20.337984 22644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:20.342885 22644 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:39027
I20260928 14:53:20.342950 22721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:39027 every 8 connection(s)
I20260928 14:53:20.343336 22644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:53:20.344184 22722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:20.346462 22722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7: Bootstrap starting.
I20260928 14:53:20.347121 22722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:20.347369 22722 log.cc:824] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:20.348047 22722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7: No bootstrap required, opened a new log
I20260928 14:53:20.350028 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 22644
I20260928 14:53:20.350132 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:53:20.350121 22722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719aa42e8b754f3d8de1fd8b77b052f7" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39027 } }
I20260928 14:53:20.350318 20946 external_mini_cluster.cc:1428] Setting key 2eb8d1605c32278a8554e3ea80d83ba1
I20260928 14:53:20.350337 22722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:20.350365 22722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 719aa42e8b754f3d8de1fd8b77b052f7, State: Initialized, Role: FOLLOWER
I20260928 14:53:20.350591 22722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719aa42e8b754f3d8de1fd8b77b052f7" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39027 } }
I20260928 14:53:20.350692 22722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:53:20.350730 22722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:53:20.350785 22722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:20.351459 22722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719aa42e8b754f3d8de1fd8b77b052f7" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39027 } }
I20260928 14:53:20.351583 22722 leader_election.cc:302] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 719aa42e8b754f3d8de1fd8b77b052f7; no voters:
I20260928 14:53:20.351735 22722 leader_election.cc:288] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:53:20.351787 22727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:20.351938 22727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7 [term 1 LEADER]: Becoming Leader. State: Replica: 719aa42e8b754f3d8de1fd8b77b052f7, State: Running, Role: LEADER
I20260928 14:53:20.352022 22722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:20.352056 22727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719aa42e8b754f3d8de1fd8b77b052f7" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39027 } }
I20260928 14:53:20.352442 22727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 719aa42e8b754f3d8de1fd8b77b052f7. Latest consensus state: current_term: 1 leader_uuid: "719aa42e8b754f3d8de1fd8b77b052f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719aa42e8b754f3d8de1fd8b77b052f7" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39027 } } }
I20260928 14:53:20.352519 22727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7 [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:20.352690 22729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "719aa42e8b754f3d8de1fd8b77b052f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719aa42e8b754f3d8de1fd8b77b052f7" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39027 } } }
I20260928 14:53:20.352756 22729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7 [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:20.352860 22734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:53:20.353355 22734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:53:20.355124 22734 catalog_manager.cc:1347] Generated new cluster ID: 9a977b20a86743c2880b807bc226234c
I20260928 14:53:20.355175 22734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:53:20 dist-test-slave-pxl6 krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:20.363022 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:20.350974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54050 (local address 127.20.116.190:39027)
0928 14:53:20.351195 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:20.351207 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:20.351249 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:53:20.351367 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:20.351374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:20.351434 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:20.351591 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:20.352442 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.354033 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.355138 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.355757 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.355860 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:20.358676 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:20.358680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:20.358681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:20.358692 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:20.361284 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:20.361702 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:20.361705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:20.361705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:20.361826 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:20.362058 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:20.362059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:20.362059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:20.362356 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:20.362500 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:20.362672 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 14:53:20.362808 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":107,"threads_started":1}
I20260928 14:53:20.363539 22734 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:53:20.364133 22734 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:53:20.373677 22734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7: Generated new TSK 0
I20260928 14:53:20.373874 22734 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
I20260928 14:53:20.406296 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:39027
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:39377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:20.497565 22749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:20.497780 22749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:20.497807 22749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:20.498955 22749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:20.498993 22749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:20.499017 22749 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 14:53:20.499030 22749 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:53:20.499047 22749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:53:20.500465 22749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:39377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.116.190:39027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:20.500695 22749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:20.500893 22749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:20.506114 22755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:20.506108 22754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:20.506114 22757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:20.506343 22749 server_base.cc:1034] running on GCE node
I20260928 14:53:20.506584 22749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:20.506790 22749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:20.507961 22749 hybrid_clock.cc:648] HybridClock initialized: now 1790607200507943 us; error 32 us; skew 500 ppm
Sep 28 14:53:20 dist-test-slave-pxl6 krb5kdc[22631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607200, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:20.510649 22749 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:53:20.511176 22749 webserver.cc:460] Webserver started at http://127.20.116.129:46309/ using document root <none> and password file <none>
I20260928 14:53:20.511451 22749 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:20.511495 22749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:20.511656 22749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:20.512777 22749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "a236b1e31f374fdcae1e1ced5829b638"
format_stamp: "Formatted at 2026-09-28 14:53:20 on dist-test-slave-pxl6"
server_key: "8f7bfee2352c41b9350c666a982ed033"
server_key_iv: "82f715bcc6761c2671906cadbc7f2c46"
server_key_version: "encryptionkey@0"
I20260928 14:53:20.513160 22749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "a236b1e31f374fdcae1e1ced5829b638"
format_stamp: "Formatted at 2026-09-28 14:53:20 on dist-test-slave-pxl6"
server_key: "8f7bfee2352c41b9350c666a982ed033"
server_key_iv: "82f715bcc6761c2671906cadbc7f2c46"
server_key_version: "encryptionkey@0"
I20260928 14:53:20.514596 22749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:53:20.515385 22764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:20.515580 22749 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:53:20.515671 22749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "a236b1e31f374fdcae1e1ced5829b638"
format_stamp: "Formatted at 2026-09-28 14:53:20 on dist-test-slave-pxl6"
server_key: "8f7bfee2352c41b9350c666a982ed033"
server_key_iv: "82f715bcc6761c2671906cadbc7f2c46"
server_key_version: "encryptionkey@0"
I20260928 14:53:20.515745 22749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:20.536793 22749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:20.538884 22749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:20.539063 22749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:20.539688 22749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:20.539728 22749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:20.539775 22749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:20.539806 22749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:53:20 dist-test-slave-pxl6 krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607200, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:20.547358 22749 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:32831
I20260928 14:53:20.547489 22880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:32831 every 8 connection(s)
I20260928 14:53:20.547854 22749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:53:20.550516 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:20.540383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:57505 (local address 127.20.116.190:39027)
0928 14:53:20.540457 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:20.540459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:20.540751 (+ 292us) server_negotiation.cc:407] Connection header received
0928 14:53:20.541488 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:20.541490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:20.541516 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:20.541613 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:20.542634 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.543520 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.544640 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.544868 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.544963 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:20.547667 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:20.547669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:20.547669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:20.547678 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:20.549215 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:20.549622 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:20.549623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:20.549623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:20.549679 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:20.549950 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:20.549951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:20.549951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:20.550063 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:20.550116 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:20.550373 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 14:53:20.550445 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:53:20.550806 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:20.540395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:39027 (local address 127.20.116.129:57505)
0928 14:53:20.540594 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:20.540657 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:20.541329 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:20.541688 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:20.541693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:20.542110 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:20.542468 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:20.542476 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.543643 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.543644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:20.544530 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:20.544533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.544991 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.544992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:20.545137 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:20.545832 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:20.545843 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:20.547518 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:20.549291 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:20.549298 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:20.549300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:20.549549 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:20.549769 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:20.549770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:20.549770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:20.549886 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:20.550122 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:20.550128 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:20.550258 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 14:53:20.550346 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 14:53:20.550485 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":106,"threads_started":1}
I20260928 14:53:20.551349 22881 heartbeater.cc:347] Connected to a master server at 127.20.116.190:39027
I20260928 14:53:20.551491 22881 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:20.551703 22881 heartbeater.cc:511] Master 127.20.116.190:39027 requested a full tablet report, sending...
I20260928 14:53:20.551949 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 22749
I20260928 14:53:20.552047 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:53:20.552235 20946 external_mini_cluster.cc:1428] Setting key a551d4c81f066b931f264c40b204fa19
I20260928 14:53:20.552285 22675 ts_manager.cc:194] Registered new tserver with Master: a236b1e31f374fdcae1e1ced5829b638 (127.20.116.129:32831)
I20260928 14:53:20.553198 22675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:57505
I20260928 14:53:20.558722 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:20.552661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:39027 (local address 127.20.116.129:42891)
0928 14:53:20.552793 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:20.552804 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:20.552860 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:20.553029 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:20.553030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:20.553244 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:20.553315 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:20.553317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.554093 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.554095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:20.554735 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:20.554737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.555122 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.555130 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:20.555204 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:20.555680 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:20.555685 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:20.555982 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:20.557624 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:20.557626 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:20.557626 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:20.557807 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:20.558024 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:20.558024 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:20.558025 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:20.558083 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:20.558360 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:20.558362 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:20.558410 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 14:53:20.558494 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 14:53:20.558619 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260928 14:53:20.558715 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:20.552648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:42891 (local address 127.20.116.190:39027)
0928 14:53:20.552692 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:20.552694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:20.552847 (+ 153us) server_negotiation.cc:407] Connection header received
0928 14:53:20.552917 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:20.552918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:20.552928 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:20.552985 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:20.553396 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.553997 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.554830 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.555040 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.555113 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:20.556085 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:20.556086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:20.556087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:20.556093 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:20.557522 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:20.557894 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:20.557895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:20.557896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:20.557950 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:20.558173 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:20.558174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:20.558174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:20.558287 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:20.558359 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:20.558521 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 14:53:20.558619 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.20.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
I20260928 14:53:20.577955 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:39027
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:39377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:20.661666 22891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:20.661906 22891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:20.661948 22891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:20.663129 22891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:20.663194 22891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:20.663228 22891 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 14:53:20.663259 22891 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:53:20.663292 22891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:53:20.664759 22891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:39377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.116.190:39027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:20.665030 22891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:20.665273 22891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:20.670403 22899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:20.670431 22896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:20.670503 22891 server_base.cc:1034] running on GCE node
W20260928 14:53:20.670441 22897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:20.670884 22891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:20.671195 22891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:20.672377 22891 hybrid_clock.cc:648] HybridClock initialized: now 1790607200672370 us; error 47 us; skew 500 ppm
Sep 28 14:53:20 dist-test-slave-pxl6 krb5kdc[22631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607200, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:20.675048 22891 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:53:20.675527 22891 webserver.cc:460] Webserver started at http://127.20.116.130:43355/ using document root <none> and password file <none>
I20260928 14:53:20.675802 22891 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:20.675879 22891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:20.676038 22891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:20.677246 22891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "371e7f54fd044a1ea8a74b65877495f8"
format_stamp: "Formatted at 2026-09-28 14:53:20 on dist-test-slave-pxl6"
server_key: "c763e59658ee146555a2c03d6c1fd8b0"
server_key_iv: "e9f2d7104395ed6d9a66f54deebef4a2"
server_key_version: "encryptionkey@0"
I20260928 14:53:20.677654 22891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "371e7f54fd044a1ea8a74b65877495f8"
format_stamp: "Formatted at 2026-09-28 14:53:20 on dist-test-slave-pxl6"
server_key: "c763e59658ee146555a2c03d6c1fd8b0"
server_key_iv: "e9f2d7104395ed6d9a66f54deebef4a2"
server_key_version: "encryptionkey@0"
I20260928 14:53:20.679172 22891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:53:20.679970 22906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:20.680189 22891 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:20.680306 22891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "371e7f54fd044a1ea8a74b65877495f8"
format_stamp: "Formatted at 2026-09-28 14:53:20 on dist-test-slave-pxl6"
server_key: "c763e59658ee146555a2c03d6c1fd8b0"
server_key_iv: "e9f2d7104395ed6d9a66f54deebef4a2"
server_key_version: "encryptionkey@0"
I20260928 14:53:20.680388 22891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:20.687956 22891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:20.690111 22891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:20.690282 22891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:20.690912 22891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:20.690977 22891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:20.691026 22891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:20.691058 22891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:53:20 dist-test-slave-pxl6 krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607200, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:20.699126 22891 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:40537
I20260928 14:53:20.699209 23021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:40537 every 8 connection(s)
I20260928 14:53:20.699594 22891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:53:20.701931 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:20.691407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:48663 (local address 127.20.116.190:39027)
0928 14:53:20.691480 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:20.691483 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:20.691671 (+ 188us) server_negotiation.cc:407] Connection header received
0928 14:53:20.692162 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:20.692163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:20.692181 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:20.692263 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:20.693136 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.694081 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.695780 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.696171 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.696264 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:20.698899 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:20.698901 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:20.698902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:20.698911 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:20.700638 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:20.701007 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:20.701008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:20.701008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:20.701113 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:20.701332 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:20.701333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:20.701333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:20.701447 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:20.701502 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:20.701795 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 14:53:20.701862 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:53:20.702065 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:20.691375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:39027 (local address 127.20.116.130:48663)
0928 14:53:20.691535 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:20.691575 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:20.692059 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:20.692351 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:20.692354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:20.692693 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:20.692987 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:20.692994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.694196 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.694197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:20.695666 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:20.695670 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.696283 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.696285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:20.696457 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:20.697193 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:20.697202 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:20.698779 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:20.700713 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:20.700717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:20.700719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:20.700932 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:20.701175 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:20.701176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:20.701176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:20.701273 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:20.701504 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:20.701508 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:20.701673 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 14:53:20.701761 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 14:53:20.701882 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260928 14:53:20.702737 23022 heartbeater.cc:347] Connected to a master server at 127.20.116.190:39027
I20260928 14:53:20.702847 23022 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:20.702857 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 22891
I20260928 14:53:20.702953 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:53:20.703063 23022 heartbeater.cc:511] Master 127.20.116.190:39027 requested a full tablet report, sending...
I20260928 14:53:20.703181 20946 external_mini_cluster.cc:1428] Setting key ed49cfbc72c43e4f7f88ea174635f29a
I20260928 14:53:20.703495 22676 ts_manager.cc:194] Registered new tserver with Master: 371e7f54fd044a1ea8a74b65877495f8 (127.20.116.130:40537)
I20260928 14:53:20.704094 22676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:48663
I20260928 14:53:20.711176 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:20.704238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:60747 (local address 127.20.116.190:39027)
0928 14:53:20.704277 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:20.704279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:20.704287 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:53:20.704306 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:20.704306 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:20.704315 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:20.704375 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:20.705267 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.705721 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.706918 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.707102 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.707161 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:20.708339 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:20.708341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:20.708341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:20.708346 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:20.709767 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:20.710195 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:20.710196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:20.710196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:20.710247 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:20.710491 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:20.710492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:20.710492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:20.710723 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:20.710779 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:20.710982 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 14:53:20.711070 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:53:20.711234 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:20.703690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:39027 (local address 127.20.116.130:60747)
0928 14:53:20.703898 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:20.703909 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:20.703953 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:20.704399 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:20.704400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:20.705075 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:20.705171 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:20.705174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.705824 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.705825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:20.706837 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:20.706841 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.707184 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.707185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:20.707280 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:20.707773 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:20.707780 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:20.708251 (+ 471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:20.709860 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:20.709861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:20.709861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:20.710110 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:20.710323 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:20.710324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:20.710325 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:20.710413 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:20.710797 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:20.710798 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:20.710858 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 14:53:20.710968 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 14:53:20.711095 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":144,"threads_started":1}
WARNING: no policy specified for kudu/127.20.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
I20260928 14:53:20.734195 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:39027
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:39377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:20.818353 23032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:20.818645 23032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:20.818691 23032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:20.819924 23032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:20.820006 23032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:20.820042 23032 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 14:53:20.820070 23032 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:53:20.820106 23032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:53:20.821668 23032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:39377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.116.190:39027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:20.821959 23032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:20.822202 23032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:20.827448 23038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:20.827458 23037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:20.827458 23040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:20.827709 23032 server_base.cc:1034] running on GCE node
I20260928 14:53:20.827921 23032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:20.828193 23032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:20.829360 23032 hybrid_clock.cc:648] HybridClock initialized: now 1790607200829347 us; error 32 us; skew 500 ppm
Sep 28 14:53:20 dist-test-slave-pxl6 krb5kdc[22631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607200, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:20.832095 23032 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:53:20.832609 23032 webserver.cc:460] Webserver started at http://127.20.116.131:42433/ using document root <none> and password file <none>
I20260928 14:53:20.832856 23032 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:20.832903 23032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:20.833060 23032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:20.834208 23032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "edc88735caac449ab12811d019c3061f"
format_stamp: "Formatted at 2026-09-28 14:53:20 on dist-test-slave-pxl6"
server_key: "d07f401175d56924ca14c77b78bc6c13"
server_key_iv: "9e3f32238f3c060233cc0008943ff40d"
server_key_version: "encryptionkey@0"
I20260928 14:53:20.834640 23032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "edc88735caac449ab12811d019c3061f"
format_stamp: "Formatted at 2026-09-28 14:53:20 on dist-test-slave-pxl6"
server_key: "d07f401175d56924ca14c77b78bc6c13"
server_key_iv: "9e3f32238f3c060233cc0008943ff40d"
server_key_version: "encryptionkey@0"
I20260928 14:53:20.836073 23032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:20.836966 23047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:20.837180 23032 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:20.837273 23032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "edc88735caac449ab12811d019c3061f"
format_stamp: "Formatted at 2026-09-28 14:53:20 on dist-test-slave-pxl6"
server_key: "d07f401175d56924ca14c77b78bc6c13"
server_key_iv: "9e3f32238f3c060233cc0008943ff40d"
server_key_version: "encryptionkey@0"
I20260928 14:53:20.837335 23032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:20.844875 23032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:20.847046 23032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:20.847165 23032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:20.847640 23032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:20.847681 23032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:20.847707 23032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:20.847759 23032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:20.855181 23032 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:38345
I20260928 14:53:20.855293 23162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:38345 every 8 connection(s)
Sep 28 14:53:20 dist-test-slave-pxl6 krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607200, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:20.855631 23032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:53:20.859001 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:20.848163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:45587 (local address 127.20.116.190:39027)
0928 14:53:20.848234 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:20.848237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:20.848440 (+ 203us) server_negotiation.cc:407] Connection header received
0928 14:53:20.848952 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:20.848954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:20.848970 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:20.849057 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:20.849982 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.850862 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.852536 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.852872 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.852962 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:20.856126 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:20.856128 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:20.856129 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:20.856137 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:20.857506 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:20.857970 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:20.857971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:20.857971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:20.858026 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:20.858301 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:20.858302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:20.858302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:20.858411 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:20.858463 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:20.858863 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 14:53:20.858934 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 14:53:20.859210 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:20.848147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:39027 (local address 127.20.116.131:45587)
0928 14:53:20.848304 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:20.848346 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:20.848849 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:20.849169 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:20.849174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:20.849551 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:20.849868 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:20.849876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.850968 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.850969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:20.852453 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:20.852457 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.852999 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.853001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:20.853232 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:20.854016 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:20.854027 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:20.856011 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:20.857630 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:20.857637 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:20.857639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:20.857898 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:20.858107 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:20.858108 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:20.858108 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:20.858228 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:20.858473 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:20.858478 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:20.858737 (+ 259us) client_negotiation.cc:769] Sending connection context
0928 14:53:20.858834 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 14:53:20.858969 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260928 14:53:20.859335 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 23032
I20260928 14:53:20.859440 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:53:20.859737 20946 external_mini_cluster.cc:1428] Setting key fa556a3b5fff430ee03eed5152964639
I20260928 14:53:20.859850 23163 heartbeater.cc:347] Connected to a master server at 127.20.116.190:39027
I20260928 14:53:20.859961 23163 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:20.860157 23163 heartbeater.cc:511] Master 127.20.116.190:39027 requested a full tablet report, sending...
I20260928 14:53:20.860564 22675 ts_manager.cc:194] Registered new tserver with Master: edc88735caac449ab12811d019c3061f (127.20.116.131:38345)
I20260928 14:53:20.861099 22675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:45587
I20260928 14:53:20.866950 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:20.861021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:59553 (local address 127.20.116.190:39027)
0928 14:53:20.861074 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:20.861076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:20.861254 (+ 178us) server_negotiation.cc:407] Connection header received
0928 14:53:20.861288 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:20.861289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:20.861301 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:20.861375 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:20.861668 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.862351 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.863196 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.863397 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.863477 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:20.864419 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:20.864421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:20.864422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:20.864429 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:20.865716 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:20.866106 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:20.866108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:20.866108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:20.866177 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:20.866375 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:20.866376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:20.866377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:20.866493 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:20.866598 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:20.866766 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 14:53:20.866848 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 14:53:20.866961 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:20.860931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:39027 (local address 127.20.116.131:59553)
0928 14:53:20.861151 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:20.861165 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:20.861236 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:20.861386 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:20.861387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:20.861550 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:20.861611 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:20.861613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.862458 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.862458 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:20.863075 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:20.863078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.863479 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.863480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:20.863561 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:20.864015 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:20.864019 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:20.864326 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:20.865826 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:20.865828 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:20.865828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:20.866014 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:20.866244 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:20.866245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:20.866245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:20.866309 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:20.866595 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:20.866596 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:20.866647 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 14:53:20.866738 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 14:53:20.866851 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":105,"threads_started":1}
I20260928 14:53:20.872100 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:53:20 dist-test-slave-pxl6 krb5kdc[22631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:53:20 dist-test-slave-pxl6 krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:20.894822 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:20.887865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54056 (local address 127.20.116.190:39027)
0928 14:53:20.887939 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:20.887941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:20.887951 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:53:20.887986 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:20.887987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:20.887997 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:20.888047 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:20.888604 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.889126 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.889849 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.890028 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.890096 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:20.892438 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:20.892440 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:20.892440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:20.892447 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:20.893765 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:20.894110 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:20.894110 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:20.894111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:20.894164 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:20.894329 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:20.894330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:20.894330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:20.894434 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:20.894509 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:20.894683 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 14:53:20.894752 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:53:20.896605 22675 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:53:20.898404 22675 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:53:20.905421 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:20.901094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:32831 (local address 127.0.0.1:57046)
0928 14:53:20.901215 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:20.901224 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:20.901267 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:20.901574 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:20.901574 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:20.901576 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:20.901638 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:20.901639 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.902989 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.902990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:20.904345 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:20.904348 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.905072 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.905073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:20.905166 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:20.905181 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 14:53:20.905250 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 14:53:20.905296 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":49,"thread_start_us":86,"threads_started":1}
I20260928 14:53:20.905624 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:20.901000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39336 (local address 127.20.116.130:40537)
0928 14:53:20.901291 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:20.901296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:20.901307 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:53:20.901347 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:20.901350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:20.901367 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:20.901482 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:20.901661 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.902925 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.904249 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.904987 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.905355 (+ 368us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:20.905449 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:20.905474 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 14:53:20.905513 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":102,"threads_started":1}
I20260928 14:53:20.905620 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:20.901460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34024 (local address 127.20.116.131:38345)
0928 14:53:20.901705 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:20.901710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:20.901718 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:53:20.901760 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:20.901761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:20.901775 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:20.901848 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:20.902071 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.903009 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.904436 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.905165 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.905248 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:20.905302 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:20.905466 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 14:53:20.905506 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":83,"threads_started":1}
I20260928 14:53:20.905954 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:20.901196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.20.116.129:32831)
0928 14:53:20.901382 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:20.901387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:20.901397 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:53:20.901440 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:20.901442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:20.901457 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:20.901588 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:20.901727 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.902888 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.904459 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.904961 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.905072 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:20.905136 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:20.905789 (+ 653us) server_negotiation.cc:299] Negotiation successful
0928 14:53:20.905849 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":71,"threads_started":1}
I20260928 14:53:20.905426 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:20.900911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:40537 (local address 127.0.0.1:39336)
0928 14:53:20.901030 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:20.901074 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:20.901145 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:20.901486 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:20.901489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:20.901491 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:20.901572 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:20.901574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.902992 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.902993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:20.904151 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:20.904154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.905110 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.905112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:20.905178 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:20.905183 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:53:20.905232 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 14:53:20.905297 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":37,"thread_start_us":34,"threads_started":1}
I20260928 14:53:20.905531 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:20.901270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:38345 (local address 127.0.0.1:34024)
0928 14:53:20.901360 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:20.901368 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:20.901405 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:20.901915 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:20.901916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:20.901919 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:20.901968 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:20.901969 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.903092 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.903092 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:20.904346 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:20.904348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:20.905245 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:20.905246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:20.905299 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:20.905302 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:53:20.905354 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 14:53:20.905391 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":99,"thread_start_us":63,"threads_started":1}
I20260928 14:53:20.906929 22956 tablet_service.cc:1467] Processing CreateTablet for tablet 5f67640289134f4485c343e5c1dcb3e3 (DEFAULT_TABLE table=test-table [id=c7757d0e6afa47b8a326ecf4160ca247]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:20.907115 23097 tablet_service.cc:1467] Processing CreateTablet for tablet 5f67640289134f4485c343e5c1dcb3e3 (DEFAULT_TABLE table=test-table [id=c7757d0e6afa47b8a326ecf4160ca247]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:20.907196 22815 tablet_service.cc:1467] Processing CreateTablet for tablet 5f67640289134f4485c343e5c1dcb3e3 (DEFAULT_TABLE table=test-table [id=c7757d0e6afa47b8a326ecf4160ca247]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:20.907372 22956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f67640289134f4485c343e5c1dcb3e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:20.907507 23097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f67640289134f4485c343e5c1dcb3e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:20.907536 22815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f67640289134f4485c343e5c1dcb3e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:20.910342 23187 tablet_bootstrap.cc:492] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f: Bootstrap starting.
I20260928 14:53:20.911362 23187 tablet_bootstrap.cc:654] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:20.911365 23188 tablet_bootstrap.cc:492] T 5f67640289134f4485c343e5c1dcb3e3 P 371e7f54fd044a1ea8a74b65877495f8: Bootstrap starting.
I20260928 14:53:20.911842 23187 log.cc:824] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:20.912038 23188 tablet_bootstrap.cc:654] T 5f67640289134f4485c343e5c1dcb3e3 P 371e7f54fd044a1ea8a74b65877495f8: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:20.912385 23188 log.cc:824] T 5f67640289134f4485c343e5c1dcb3e3 P 371e7f54fd044a1ea8a74b65877495f8: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:20.912673 23187 tablet_bootstrap.cc:492] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f: No bootstrap required, opened a new log
I20260928 14:53:20.912743 23187 ts_tablet_manager.cc:1397] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:53:20.913838 23189 tablet_bootstrap.cc:492] T 5f67640289134f4485c343e5c1dcb3e3 P a236b1e31f374fdcae1e1ced5829b638: Bootstrap starting.
I20260928 14:53:20.914476 23189 tablet_bootstrap.cc:654] T 5f67640289134f4485c343e5c1dcb3e3 P a236b1e31f374fdcae1e1ced5829b638: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:20.914861 23189 log.cc:824] T 5f67640289134f4485c343e5c1dcb3e3 P a236b1e31f374fdcae1e1ced5829b638: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:20.915088 23187 raft_consensus.cc:348] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:20.915199 23188 tablet_bootstrap.cc:492] T 5f67640289134f4485c343e5c1dcb3e3 P 371e7f54fd044a1ea8a74b65877495f8: No bootstrap required, opened a new log
I20260928 14:53:20.915269 23187 raft_consensus.cc:374] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:20.915298 23188 ts_tablet_manager.cc:1397] T 5f67640289134f4485c343e5c1dcb3e3 P 371e7f54fd044a1ea8a74b65877495f8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 14:53:20.915311 23187 raft_consensus.cc:729] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc88735caac449ab12811d019c3061f, State: Initialized, Role: FOLLOWER
I20260928 14:53:20.915422 23187 consensus_queue.cc:260] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:20.915588 23189 tablet_bootstrap.cc:492] T 5f67640289134f4485c343e5c1dcb3e3 P a236b1e31f374fdcae1e1ced5829b638: No bootstrap required, opened a new log
I20260928 14:53:20.915676 23189 ts_tablet_manager.cc:1397] T 5f67640289134f4485c343e5c1dcb3e3 P a236b1e31f374fdcae1e1ced5829b638: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:53:20.915685 23187 ts_tablet_manager.cc:1428] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:53:20.915810 23163 heartbeater.cc:503] Master 127.20.116.190:39027 was elected leader, sending a full tablet report...
I20260928 14:53:20.917490 23188 raft_consensus.cc:348] T 5f67640289134f4485c343e5c1dcb3e3 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:20.917670 23188 raft_consensus.cc:374] T 5f67640289134f4485c343e5c1dcb3e3 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:20.917704 23188 raft_consensus.cc:729] T 5f67640289134f4485c343e5c1dcb3e3 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371e7f54fd044a1ea8a74b65877495f8, State: Initialized, Role: FOLLOWER
I20260928 14:53:20.917827 23188 consensus_queue.cc:260] T 5f67640289134f4485c343e5c1dcb3e3 P 371e7f54fd044a1ea8a74b65877495f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:20.917865 23189 raft_consensus.cc:348] T 5f67640289134f4485c343e5c1dcb3e3 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:20.918016 23189 raft_consensus.cc:374] T 5f67640289134f4485c343e5c1dcb3e3 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:20.918067 23189 raft_consensus.cc:729] T 5f67640289134f4485c343e5c1dcb3e3 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a236b1e31f374fdcae1e1ced5829b638, State: Initialized, Role: FOLLOWER
I20260928 14:53:20.918167 23188 ts_tablet_manager.cc:1428] T 5f67640289134f4485c343e5c1dcb3e3 P 371e7f54fd044a1ea8a74b65877495f8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:53:20.918179 23189 consensus_queue.cc:260] T 5f67640289134f4485c343e5c1dcb3e3 P a236b1e31f374fdcae1e1ced5829b638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:20.918221 23022 heartbeater.cc:503] Master 127.20.116.190:39027 was elected leader, sending a full tablet report...
I20260928 14:53:20.918434 23189 ts_tablet_manager.cc:1428] T 5f67640289134f4485c343e5c1dcb3e3 P a236b1e31f374fdcae1e1ced5829b638: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:53:20.918503 22881 heartbeater.cc:503] Master 127.20.116.190:39027 was elected leader, sending a full tablet report...
W20260928 14:53:20.950258 23023 tablet.cc:2369] T 5f67640289134f4485c343e5c1dcb3e3 P 371e7f54fd044a1ea8a74b65877495f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:53:21.049080 22882 tablet.cc:2369] T 5f67640289134f4485c343e5c1dcb3e3 P a236b1e31f374fdcae1e1ced5829b638: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:53:21.106248 23164 tablet.cc:2369] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:53:21.282823 23193 raft_consensus.cc:482] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:53:21.282974 23193 raft_consensus.cc:504] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:21.283315 23193 leader_election.cc:288] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371e7f54fd044a1ea8a74b65877495f8 (127.20.116.130:40537), a236b1e31f374fdcae1e1ced5829b638 (127.20.116.129:32831)
I20260928 14:53:21.286682 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:21.283746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:60839 (local address 127.20.116.130:40537)
0928 14:53:21.283823 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:21.283825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:21.283837 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:53:21.283878 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:21.283879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:21.283889 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:21.283966 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:21.284170 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.284806 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.285776 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.286266 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.286350 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:21.286360 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:21.286525 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 14:53:21.286601 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:53:21.286686 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:21.283627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:40537 (local address 127.20.116.131:60839)
0928 14:53:21.283697 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:21.283709 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:21.283762 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:21.283989 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:21.283991 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:21.283993 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:21.284073 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:21.284075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.284898 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.284899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:21.285672 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:21.285674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.286348 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.286349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:21.286440 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:21.286443 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:53:21.286506 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 14:53:21.286611 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 14:53:21.286693 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:21.283863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:43427 (local address 127.20.116.129:32831)
0928 14:53:21.283907 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:21.283909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:21.284014 (+ 105us) server_negotiation.cc:407] Connection header received
0928 14:53:21.284082 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:21.284083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:21.284093 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:21.284168 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:21.284350 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.284887 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.285810 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.286266 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.286351 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:21.286361 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:21.286565 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 14:53:21.286626 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:53:21.286701 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:21.283759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:32831 (local address 127.20.116.131:43427)
0928 14:53:21.283925 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:21.283934 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:21.284022 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:21.284172 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:21.284173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:21.284174 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:21.284251 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:21.284253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.284975 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.284976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:21.285713 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:21.285715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.286351 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.286352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:21.286429 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:21.286433 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:53:21.286524 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 14:53:21.286601 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":102,"threads_started":1}
I20260928 14:53:21.287151 22976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f67640289134f4485c343e5c1dcb3e3" candidate_uuid: "edc88735caac449ab12811d019c3061f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371e7f54fd044a1ea8a74b65877495f8" is_pre_election: true
I20260928 14:53:21.287357 22976 raft_consensus.cc:2454] T 5f67640289134f4485c343e5c1dcb3e3 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc88735caac449ab12811d019c3061f in term 0.
I20260928 14:53:21.287348 22835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f67640289134f4485c343e5c1dcb3e3" candidate_uuid: "edc88735caac449ab12811d019c3061f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a236b1e31f374fdcae1e1ced5829b638" is_pre_election: true
I20260928 14:53:21.287514 22835 raft_consensus.cc:2454] T 5f67640289134f4485c343e5c1dcb3e3 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc88735caac449ab12811d019c3061f in term 0.
I20260928 14:53:21.287628 23051 leader_election.cc:302] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371e7f54fd044a1ea8a74b65877495f8, edc88735caac449ab12811d019c3061f; no voters:
I20260928 14:53:21.287773 23193 raft_consensus.cc:2781] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:53:21.287833 23193 raft_consensus.cc:482] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:53:21.287858 23193 raft_consensus.cc:3037] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:21.288852 23193 raft_consensus.cc:504] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:21.289009 23193 leader_election.cc:288] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f [CANDIDATE]: Term 1 election: Requested vote from peers 371e7f54fd044a1ea8a74b65877495f8 (127.20.116.130:40537), a236b1e31f374fdcae1e1ced5829b638 (127.20.116.129:32831)
I20260928 14:53:21.289237 22835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f67640289134f4485c343e5c1dcb3e3" candidate_uuid: "edc88735caac449ab12811d019c3061f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a236b1e31f374fdcae1e1ced5829b638"
I20260928 14:53:21.289328 22835 raft_consensus.cc:3037] T 5f67640289134f4485c343e5c1dcb3e3 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:21.289999 22835 raft_consensus.cc:2454] T 5f67640289134f4485c343e5c1dcb3e3 P a236b1e31f374fdcae1e1ced5829b638 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc88735caac449ab12811d019c3061f in term 1.
I20260928 14:53:21.290105 22976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f67640289134f4485c343e5c1dcb3e3" candidate_uuid: "edc88735caac449ab12811d019c3061f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371e7f54fd044a1ea8a74b65877495f8"
I20260928 14:53:21.290220 23051 leader_election.cc:302] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a236b1e31f374fdcae1e1ced5829b638, edc88735caac449ab12811d019c3061f; no voters:
I20260928 14:53:21.290221 22976 raft_consensus.cc:3037] T 5f67640289134f4485c343e5c1dcb3e3 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:21.290357 23193 raft_consensus.cc:2781] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:21.290583 23193 raft_consensus.cc:686] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f [term 1 LEADER]: Becoming Leader. State: Replica: edc88735caac449ab12811d019c3061f, State: Running, Role: LEADER
I20260928 14:53:21.290714 23193 consensus_queue.cc:237] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:21.291173 22976 raft_consensus.cc:2454] T 5f67640289134f4485c343e5c1dcb3e3 P 371e7f54fd044a1ea8a74b65877495f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc88735caac449ab12811d019c3061f in term 1.
I20260928 14:53:21.291740 22675 catalog_manager.cc:5495] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f reported cstate change: term changed from 0 to 1, leader changed from <none> to edc88735caac449ab12811d019c3061f (127.20.116.131). New cstate: current_term: 1 leader_uuid: "edc88735caac449ab12811d019c3061f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } health_report { overall_health: HEALTHY } } }
Sep 28 14:53:21 dist-test-slave-pxl6 krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607200, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:21.378656 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:21.371264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:39027 (local address 127.0.0.1:54070)
0928 14:53:21.371552 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:21.371564 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:21.371646 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:21.371782 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:21.371783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:21.372102 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:21.372188 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:21.372191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.372865 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.372866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:21.373400 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:21.373402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.373835 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.373836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:21.373880 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:21.374406 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:21.374411 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:21.375981 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:21.377619 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:21.377627 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:21.377629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:21.377856 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:21.378043 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:21.378044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:21.378044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:21.378102 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:21.378316 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:21.378322 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:21.378364 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 14:53:21.378416 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 14:53:21.378576 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":179,"threads_started":1}
I20260928 14:53:21.378736 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:21.371488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54070 (local address 127.20.116.190:39027)
0928 14:53:21.371647 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:21.371649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:21.371683 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:53:21.371707 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:21.371708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:21.371717 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:21.371759 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:21.372268 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.372789 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.373548 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.373757 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.373823 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:21.376095 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:21.376097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:21.376097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:21.376104 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:21.377530 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:21.377926 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:21.377927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:21.377927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:21.377981 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:21.378156 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:21.378157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:21.378157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:21.378262 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:21.378314 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:21.378446 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 14:53:21.378662 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":116}
I20260928 14:53:21.380852 22675 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:54070:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 14:53:21.381078 22675 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:53:21.382895 22815 tablet_service.cc:1467] Processing CreateTablet for tablet b3fb896df718483caee8992cba3ae296 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=99ea39c9e2804e7ea28b0e4b5a8b2381]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:53:21.382920 23097 tablet_service.cc:1467] Processing CreateTablet for tablet b3fb896df718483caee8992cba3ae296 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=99ea39c9e2804e7ea28b0e4b5a8b2381]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:53:21.383071 22815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3fb896df718483caee8992cba3ae296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:21.383088 23097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3fb896df718483caee8992cba3ae296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:21.384397 23189 tablet_bootstrap.cc:492] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638: Bootstrap starting.
I20260928 14:53:21.384410 23187 tablet_bootstrap.cc:492] T b3fb896df718483caee8992cba3ae296 P edc88735caac449ab12811d019c3061f: Bootstrap starting.
I20260928 14:53:21.384508 22956 tablet_service.cc:1467] Processing CreateTablet for tablet b3fb896df718483caee8992cba3ae296 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=99ea39c9e2804e7ea28b0e4b5a8b2381]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:53:21.384667 22956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3fb896df718483caee8992cba3ae296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:21.384846 23187 tablet_bootstrap.cc:654] T b3fb896df718483caee8992cba3ae296 P edc88735caac449ab12811d019c3061f: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:21.384850 23189 tablet_bootstrap.cc:654] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:21.385489 23187 tablet_bootstrap.cc:492] T b3fb896df718483caee8992cba3ae296 P edc88735caac449ab12811d019c3061f: No bootstrap required, opened a new log
I20260928 14:53:21.385521 23189 tablet_bootstrap.cc:492] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638: No bootstrap required, opened a new log
I20260928 14:53:21.385564 23189 ts_tablet_manager.cc:1397] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:21.385563 23187 ts_tablet_manager.cc:1397] T b3fb896df718483caee8992cba3ae296 P edc88735caac449ab12811d019c3061f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:21.385710 23187 raft_consensus.cc:348] T b3fb896df718483caee8992cba3ae296 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:21.385756 23189 raft_consensus.cc:348] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:21.385797 23187 raft_consensus.cc:374] T b3fb896df718483caee8992cba3ae296 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:21.385826 23187 raft_consensus.cc:729] T b3fb896df718483caee8992cba3ae296 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc88735caac449ab12811d019c3061f, State: Initialized, Role: FOLLOWER
I20260928 14:53:21.385828 23189 raft_consensus.cc:374] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:21.385854 23189 raft_consensus.cc:729] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a236b1e31f374fdcae1e1ced5829b638, State: Initialized, Role: FOLLOWER
I20260928 14:53:21.385875 23187 consensus_queue.cc:260] T b3fb896df718483caee8992cba3ae296 P edc88735caac449ab12811d019c3061f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:21.385969 23187 ts_tablet_manager.cc:1428] T b3fb896df718483caee8992cba3ae296 P edc88735caac449ab12811d019c3061f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:21.386013 23193 tablet_replica.cc:439] T b3fb896df718483caee8992cba3ae296 P edc88735caac449ab12811d019c3061f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } } }
I20260928 14:53:21.386106 23193 tablet_replica.cc:442] T b3fb896df718483caee8992cba3ae296 P edc88735caac449ab12811d019c3061f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:53:21.386291 23188 tablet_bootstrap.cc:492] T b3fb896df718483caee8992cba3ae296 P 371e7f54fd044a1ea8a74b65877495f8: Bootstrap starting.
I20260928 14:53:21.386736 23188 tablet_bootstrap.cc:654] T b3fb896df718483caee8992cba3ae296 P 371e7f54fd044a1ea8a74b65877495f8: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:21.386726 23189 consensus_queue.cc:260] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:21.386909 23195 tablet_replica.cc:439] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } } }
I20260928 14:53:21.386978 23195 tablet_replica.cc:442] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:53:21.387174 23189 ts_tablet_manager.cc:1428] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:53:21.387387 23188 tablet_bootstrap.cc:492] T b3fb896df718483caee8992cba3ae296 P 371e7f54fd044a1ea8a74b65877495f8: No bootstrap required, opened a new log
I20260928 14:53:21.387437 23188 ts_tablet_manager.cc:1397] T b3fb896df718483caee8992cba3ae296 P 371e7f54fd044a1ea8a74b65877495f8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:21.387554 23188 raft_consensus.cc:348] T b3fb896df718483caee8992cba3ae296 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:21.387601 23188 raft_consensus.cc:374] T b3fb896df718483caee8992cba3ae296 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:21.387614 23188 raft_consensus.cc:729] T b3fb896df718483caee8992cba3ae296 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371e7f54fd044a1ea8a74b65877495f8, State: Initialized, Role: FOLLOWER
I20260928 14:53:21.387677 23188 consensus_queue.cc:260] T b3fb896df718483caee8992cba3ae296 P 371e7f54fd044a1ea8a74b65877495f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:21.387809 23188 ts_tablet_manager.cc:1428] T b3fb896df718483caee8992cba3ae296 P 371e7f54fd044a1ea8a74b65877495f8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:21.387861 23194 tablet_replica.cc:439] T b3fb896df718483caee8992cba3ae296 P 371e7f54fd044a1ea8a74b65877495f8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } } }
I20260928 14:53:21.387960 23194 tablet_replica.cc:442] T b3fb896df718483caee8992cba3ae296 P 371e7f54fd044a1ea8a74b65877495f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:53:21.464821 23195 raft_consensus.cc:482] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:53:21.464943 23195 raft_consensus.cc:504] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:21.465360 23195 leader_election.cc:288] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371e7f54fd044a1ea8a74b65877495f8 (127.20.116.130:40537), edc88735caac449ab12811d019c3061f (127.20.116.131:38345)
I20260928 14:53:21.469803 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:21.465668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:43159 (local address 127.20.116.130:40537)
0928 14:53:21.465745 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:21.465747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:21.465969 (+ 222us) server_negotiation.cc:407] Connection header received
0928 14:53:21.466007 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:21.466008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:21.466020 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:21.466087 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:21.466299 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.467073 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.468732 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.469297 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.469377 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:21.469387 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:21.469707 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 14:53:21.469743 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:53:21.469859 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:21.465550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:40537 (local address 127.20.116.129:43159)
0928 14:53:21.465741 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:21.465754 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:21.465856 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:21.466116 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:21.466117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:21.466120 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:21.466213 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:21.466215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.467174 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.467175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:21.468640 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:21.468642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.469448 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.469450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:21.469602 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:21.469610 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 14:53:21.469692 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 14:53:21.469753 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":80,"thread_start_us":91,"threads_started":1}
I20260928 14:53:21.470052 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:21.465778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:38345 (local address 127.20.116.129:48297)
0928 14:53:21.465877 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:21.465888 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:21.465952 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:21.466140 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:21.466140 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:21.466142 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:21.466213 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:21.466216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.467175 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.467176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:21.468618 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:21.468623 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.469757 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.469758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:21.469822 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:21.469826 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:53:21.469880 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 14:53:21.469921 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":97,"thread_start_us":55,"threads_started":1}
I20260928 14:53:21.470232 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:21.465873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:48297 (local address 127.20.116.131:38345)
0928 14:53:21.466005 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:21.466008 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:21.466016 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:53:21.466040 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:21.466041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:21.466051 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:21.466124 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:21.466300 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.467095 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.468732 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.469298 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.469378 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:21.469387 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:21.470053 (+ 666us) server_negotiation.cc:299] Negotiation successful
0928 14:53:21.470180 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
I20260928 14:53:21.470332 22976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3fb896df718483caee8992cba3ae296" candidate_uuid: "a236b1e31f374fdcae1e1ced5829b638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371e7f54fd044a1ea8a74b65877495f8" is_pre_election: true
I20260928 14:53:21.470453 22976 raft_consensus.cc:2454] T b3fb896df718483caee8992cba3ae296 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a236b1e31f374fdcae1e1ced5829b638 in term 0.
I20260928 14:53:21.470573 23117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3fb896df718483caee8992cba3ae296" candidate_uuid: "a236b1e31f374fdcae1e1ced5829b638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc88735caac449ab12811d019c3061f" is_pre_election: true
I20260928 14:53:21.470664 22768 leader_election.cc:302] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371e7f54fd044a1ea8a74b65877495f8, a236b1e31f374fdcae1e1ced5829b638; no voters:
I20260928 14:53:21.470731 23117 raft_consensus.cc:2454] T b3fb896df718483caee8992cba3ae296 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a236b1e31f374fdcae1e1ced5829b638 in term 0.
I20260928 14:53:21.470824 23195 raft_consensus.cc:2781] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:53:21.470862 23195 raft_consensus.cc:482] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:53:21.470881 23195 raft_consensus.cc:3037] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:21.471563 23195 raft_consensus.cc:504] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:21.471671 23195 leader_election.cc:288] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638 [CANDIDATE]: Term 1 election: Requested vote from peers 371e7f54fd044a1ea8a74b65877495f8 (127.20.116.130:40537), edc88735caac449ab12811d019c3061f (127.20.116.131:38345)
I20260928 14:53:21.471864 22976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3fb896df718483caee8992cba3ae296" candidate_uuid: "a236b1e31f374fdcae1e1ced5829b638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371e7f54fd044a1ea8a74b65877495f8"
I20260928 14:53:21.471957 22976 raft_consensus.cc:3037] T b3fb896df718483caee8992cba3ae296 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:21.471908 23117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3fb896df718483caee8992cba3ae296" candidate_uuid: "a236b1e31f374fdcae1e1ced5829b638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc88735caac449ab12811d019c3061f"
I20260928 14:53:21.472033 23117 raft_consensus.cc:3037] T b3fb896df718483caee8992cba3ae296 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:21.472574 22976 raft_consensus.cc:2454] T b3fb896df718483caee8992cba3ae296 P 371e7f54fd044a1ea8a74b65877495f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a236b1e31f374fdcae1e1ced5829b638 in term 1.
I20260928 14:53:21.472707 23117 raft_consensus.cc:2454] T b3fb896df718483caee8992cba3ae296 P edc88735caac449ab12811d019c3061f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a236b1e31f374fdcae1e1ced5829b638 in term 1.
I20260928 14:53:21.472740 22768 leader_election.cc:302] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371e7f54fd044a1ea8a74b65877495f8, a236b1e31f374fdcae1e1ced5829b638; no voters:
I20260928 14:53:21.472818 23195 raft_consensus.cc:2781] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:21.472954 23195 raft_consensus.cc:686] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638 [term 1 LEADER]: Becoming Leader. State: Replica: a236b1e31f374fdcae1e1ced5829b638, State: Running, Role: LEADER
I20260928 14:53:21.473075 23195 consensus_queue.cc:237] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:21.473500 23211 tablet_replica.cc:439] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638: TxnStatusTablet state changed. Reason: New leader a236b1e31f374fdcae1e1ced5829b638. Latest consensus state: current_term: 1 leader_uuid: "a236b1e31f374fdcae1e1ced5829b638" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } } }
I20260928 14:53:21.473592 23211 tablet_replica.cc:442] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:53:21.473781 23213 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 14:53:21.473893 23213 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 14:53:21.473917 22676 catalog_manager.cc:5495] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638 reported cstate change: term changed from 0 to 1, leader changed from <none> to a236b1e31f374fdcae1e1ced5829b638 (127.20.116.129). New cstate: current_term: 1 leader_uuid: "a236b1e31f374fdcae1e1ced5829b638" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } health_report { overall_health: UNKNOWN } } }
I20260928 14:53:21.483060 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:21.480102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:32831 (local address 127.0.0.1:57058)
0928 14:53:21.480196 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:21.480210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:21.480264 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:21.480437 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:21.480438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:21.480439 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:53:21.480486 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:21.480488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.481412 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.481412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:21.481987 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:21.481988 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.482454 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.482455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:21.482494 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:21.482510 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:53:21.482894 (+ 384us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:53:21.482904 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 14:53:21.482955 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 14:53:21.483005 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 14:53:21.483060 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:21.480223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57058 (local address 127.20.116.129:32831)
0928 14:53:21.480292 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:21.480294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:21.480304 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:53:21.480337 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:21.480338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:21.480358 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:21.480437 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:21.480530 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.481308 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.482074 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.482374 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.482451 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:21.482675 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:21.482805 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:21.482902 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:21.482960 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 14:53:21.482983 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:53:21.484905 23117 raft_consensus.cc:1261] T b3fb896df718483caee8992cba3ae296 P edc88735caac449ab12811d019c3061f [term 1 FOLLOWER]: Refusing update from remote peer a236b1e31f374fdcae1e1ced5829b638: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:21.484961 22976 raft_consensus.cc:1261] T b3fb896df718483caee8992cba3ae296 P 371e7f54fd044a1ea8a74b65877495f8 [term 1 FOLLOWER]: Refusing update from remote peer a236b1e31f374fdcae1e1ced5829b638: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:21.485224 23211 consensus_queue.cc:1035] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:21.485373 23195 consensus_queue.cc:1035] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:21.486614 23194 tablet_replica.cc:439] T b3fb896df718483caee8992cba3ae296 P 371e7f54fd044a1ea8a74b65877495f8: TxnStatusTablet state changed. Reason: New leader a236b1e31f374fdcae1e1ced5829b638. Latest consensus state: current_term: 1 leader_uuid: "a236b1e31f374fdcae1e1ced5829b638" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } } }
I20260928 14:53:21.486708 23194 tablet_replica.cc:442] T b3fb896df718483caee8992cba3ae296 P 371e7f54fd044a1ea8a74b65877495f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:53:21.486656 23193 tablet_replica.cc:439] T b3fb896df718483caee8992cba3ae296 P edc88735caac449ab12811d019c3061f: TxnStatusTablet state changed. Reason: New leader a236b1e31f374fdcae1e1ced5829b638. Latest consensus state: current_term: 1 leader_uuid: "a236b1e31f374fdcae1e1ced5829b638" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } } }
I20260928 14:53:21.486734 23193 tablet_replica.cc:442] T b3fb896df718483caee8992cba3ae296 P edc88735caac449ab12811d019c3061f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:53:21.487838 23195 tablet_replica.cc:439] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a236b1e31f374fdcae1e1ced5829b638" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } } }
I20260928 14:53:21.487923 23195 tablet_replica.cc:442] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:53:21.488054 23193 tablet_replica.cc:439] T b3fb896df718483caee8992cba3ae296 P edc88735caac449ab12811d019c3061f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a236b1e31f374fdcae1e1ced5829b638" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } } }
I20260928 14:53:21.488142 23193 tablet_replica.cc:442] T b3fb896df718483caee8992cba3ae296 P edc88735caac449ab12811d019c3061f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:53:21.488310 23217 mvcc.cc:205] Tried to move back new op lower bound from 7334327097279586304 to 7334327097234165760. Current Snapshot: MvccSnapshot[applied={T|T < 7334327097279586304}]
I20260928 14:53:21.489017 23211 tablet_replica.cc:439] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a236b1e31f374fdcae1e1ced5829b638" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } } }
I20260928 14:53:21.489101 23211 tablet_replica.cc:442] T b3fb896df718483caee8992cba3ae296 P a236b1e31f374fdcae1e1ced5829b638: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:53:21.489547 23194 tablet_replica.cc:439] T b3fb896df718483caee8992cba3ae296 P 371e7f54fd044a1ea8a74b65877495f8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a236b1e31f374fdcae1e1ced5829b638" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } } }
I20260928 14:53:21.489641 23194 tablet_replica.cc:442] T b3fb896df718483caee8992cba3ae296 P 371e7f54fd044a1ea8a74b65877495f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:53:21.495111 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:21.492255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34036 (local address 127.20.116.131:38345)
0928 14:53:21.492294 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:21.492296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:21.492399 (+ 103us) server_negotiation.cc:407] Connection header received
0928 14:53:21.492449 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:21.492450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:21.492468 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:21.492554 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:21.492649 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.493383 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.494128 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.494435 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.494519 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:21.494773 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:21.494892 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:21.494973 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:21.495019 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 14:53:21.495040 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:53:21.500557 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:21.497753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:32831 (local address 127.20.116.131:43087)
0928 14:53:21.497887 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:21.497905 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:21.497955 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:21.498077 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:21.498077 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:21.498078 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:53:21.498128 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:21.498129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.498997 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.498998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:21.499588 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:21.499591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.500144 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.500145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:21.500203 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:21.500215 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:53:21.500412 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:53:21.500414 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:53:21.500450 (+ 36us) client_negotiation.cc:240] Negotiation successful
0928 14:53:21.500482 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
I20260928 14:53:21.500554 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:21.497828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:43087 (local address 127.20.116.129:32831)
0928 14:53:21.497872 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:21.497874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:21.497948 (+ 74us) server_negotiation.cc:407] Connection header received
0928 14:53:21.497999 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:21.498000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:21.498010 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:21.498075 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:21.498167 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.498915 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.499677 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.499981 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.500054 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:21.500315 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:21.500349 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:21.500404 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:21.500455 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 14:53:21.500475 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:53:21.502959 22976 raft_consensus.cc:1261] T 5f67640289134f4485c343e5c1dcb3e3 P 371e7f54fd044a1ea8a74b65877495f8 [term 1 FOLLOWER]: Refusing update from remote peer edc88735caac449ab12811d019c3061f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:21.503068 22835 raft_consensus.cc:1261] T 5f67640289134f4485c343e5c1dcb3e3 P a236b1e31f374fdcae1e1ced5829b638 [term 1 FOLLOWER]: Refusing update from remote peer edc88735caac449ab12811d019c3061f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:21.503391 23197 consensus_queue.cc:1035] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f [LEADER]: Connected to new peer: Peer: permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:21.503520 23193 consensus_queue.cc:1035] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:21.505510 23218 mvcc.cc:205] Tried to move back new op lower bound from 7334327097354399744 to 7334327096487321600. Current Snapshot: MvccSnapshot[applied={T|T < 7334327097354399744}]
I20260928 14:53:21.506251 23216 mvcc.cc:205] Tried to move back new op lower bound from 7334327097354399744 to 7334327096487321600. Current Snapshot: MvccSnapshot[applied={T|T < 7334327097354399744}]
I20260928 14:53:21.513139 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:21.510012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:33175 (local address 127.20.116.131:38345)
0928 14:53:21.510105 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:21.510108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:21.510193 (+ 85us) server_negotiation.cc:407] Connection header received
0928 14:53:21.510251 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:21.510251 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:21.510262 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:21.510323 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:21.510419 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.511208 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.512316 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.512586 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.512675 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:21.512866 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:21.512900 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:21.512975 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:21.513039 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 14:53:21.513062 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 14:53:21.513216 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:21.509967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:38345 (local address 127.20.116.129:33175)
0928 14:53:21.510123 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:21.510141 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:21.510208 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:21.510324 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:21.510325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:21.510326 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:53:21.510376 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:21.510377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.511290 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.511291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:21.512249 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:21.512252 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:21.512676 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:21.512677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:21.512749 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:21.512774 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:53:21.512979 (+ 205us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:53:21.512982 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:53:21.513034 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 14:53:21.513075 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":97,"threads_started":1}
I20260928 14:53:21.523526 22676 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:21.523650 22676 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:21.524480 22676 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7: Sending DeleteTablet for 3 replicas of tablet 5f67640289134f4485c343e5c1dcb3e3
I20260928 14:53:21.524956 22956 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f67640289134f4485c343e5c1dcb3e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:21 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:39336
I20260928 14:53:21.524968 23097 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f67640289134f4485c343e5c1dcb3e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:21 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:34024
I20260928 14:53:21.525405 22815 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f67640289134f4485c343e5c1dcb3e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:21 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:57046
I20260928 14:53:21.525650 23239 tablet_replica.cc:330] T 5f67640289134f4485c343e5c1dcb3e3 P a236b1e31f374fdcae1e1ced5829b638: stopping tablet replica
I20260928 14:53:21.525771 23239 raft_consensus.cc:2229] T 5f67640289134f4485c343e5c1dcb3e3 P a236b1e31f374fdcae1e1ced5829b638 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:21.525823 23239 raft_consensus.cc:2258] T 5f67640289134f4485c343e5c1dcb3e3 P a236b1e31f374fdcae1e1ced5829b638 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:21.525921 22676 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:53:21.526067 22676 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:53:21.526222 23239 ts_tablet_manager.cc:1905] T 5f67640289134f4485c343e5c1dcb3e3 P a236b1e31f374fdcae1e1ced5829b638: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:21.527539 23097 tablet_service.cc:1467] Processing CreateTablet for tablet 8474ba477f0d49bb9b181b7336b66b7c (DEFAULT_TABLE table=test-table [id=b992c2e4355f4d64b26ba6bcb763c69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:21.527679 23097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8474ba477f0d49bb9b181b7336b66b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:21.528664 23239 ts_tablet_manager.cc:1918] T 5f67640289134f4485c343e5c1dcb3e3 P a236b1e31f374fdcae1e1ced5829b638: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:53:21.528733 23239 log.cc:1196] T 5f67640289134f4485c343e5c1dcb3e3 P a236b1e31f374fdcae1e1ced5829b638: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-0/wal/wals/5f67640289134f4485c343e5c1dcb3e3
I20260928 14:53:21.528991 23239 ts_tablet_manager.cc:1939] T 5f67640289134f4485c343e5c1dcb3e3 P a236b1e31f374fdcae1e1ced5829b638: Deleting consensus metadata
I20260928 14:53:21.529150 23187 tablet_bootstrap.cc:492] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f: Bootstrap starting.
I20260928 14:53:21.529381 22956 tablet_service.cc:1467] Processing CreateTablet for tablet 8474ba477f0d49bb9b181b7336b66b7c (DEFAULT_TABLE table=test-table [id=b992c2e4355f4d64b26ba6bcb763c69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:21.529495 22956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8474ba477f0d49bb9b181b7336b66b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:21.529572 23187 tablet_bootstrap.cc:654] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:21.530393 23187 tablet_bootstrap.cc:492] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f: No bootstrap required, opened a new log
I20260928 14:53:21.530435 23187 ts_tablet_manager.cc:1397] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:21.530632 23187 raft_consensus.cc:348] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:21.530699 23187 raft_consensus.cc:374] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:21.530720 23187 raft_consensus.cc:729] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc88735caac449ab12811d019c3061f, State: Initialized, Role: FOLLOWER
I20260928 14:53:21.530762 23187 consensus_queue.cc:260] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:21.530853 23187 ts_tablet_manager.cc:1428] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 14:53:21.530866 23188 tablet_bootstrap.cc:492] T 8474ba477f0d49bb9b181b7336b66b7c P 371e7f54fd044a1ea8a74b65877495f8: Bootstrap starting.
I20260928 14:53:21.531518 23188 tablet_bootstrap.cc:654] T 8474ba477f0d49bb9b181b7336b66b7c P 371e7f54fd044a1ea8a74b65877495f8: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:21.532195 23188 tablet_bootstrap.cc:492] T 8474ba477f0d49bb9b181b7336b66b7c P 371e7f54fd044a1ea8a74b65877495f8: No bootstrap required, opened a new log
I20260928 14:53:21.532243 23188 ts_tablet_manager.cc:1397] T 8474ba477f0d49bb9b181b7336b66b7c P 371e7f54fd044a1ea8a74b65877495f8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:21.532390 22815 tablet_service.cc:1467] Processing CreateTablet for tablet 8474ba477f0d49bb9b181b7336b66b7c (DEFAULT_TABLE table=test-table [id=b992c2e4355f4d64b26ba6bcb763c69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:21.532378 23188 raft_consensus.cc:348] T 8474ba477f0d49bb9b181b7336b66b7c P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:21.532444 23188 raft_consensus.cc:374] T 8474ba477f0d49bb9b181b7336b66b7c P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:21.532465 23188 raft_consensus.cc:729] T 8474ba477f0d49bb9b181b7336b66b7c P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371e7f54fd044a1ea8a74b65877495f8, State: Initialized, Role: FOLLOWER
I20260928 14:53:21.532495 22815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8474ba477f0d49bb9b181b7336b66b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:21.532518 23188 consensus_queue.cc:260] T 8474ba477f0d49bb9b181b7336b66b7c P 371e7f54fd044a1ea8a74b65877495f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:21.532601 23188 ts_tablet_manager.cc:1428] T 8474ba477f0d49bb9b181b7336b66b7c P 371e7f54fd044a1ea8a74b65877495f8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:53:21.533704 23189 tablet_bootstrap.cc:492] T 8474ba477f0d49bb9b181b7336b66b7c P a236b1e31f374fdcae1e1ced5829b638: Bootstrap starting.
I20260928 14:53:21.534014 22663 catalog_manager.cc:4841] TS a236b1e31f374fdcae1e1ced5829b638 (127.20.116.129:32831): tablet 5f67640289134f4485c343e5c1dcb3e3 (table test-table [id=c7757d0e6afa47b8a326ecf4160ca247]) successfully deleted
I20260928 14:53:21.534083 23189 tablet_bootstrap.cc:654] T 8474ba477f0d49bb9b181b7336b66b7c P a236b1e31f374fdcae1e1ced5829b638: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:21.534607 23237 tablet_replica.cc:330] T 5f67640289134f4485c343e5c1dcb3e3 P 371e7f54fd044a1ea8a74b65877495f8: stopping tablet replica
I20260928 14:53:21.534791 23237 raft_consensus.cc:2229] T 5f67640289134f4485c343e5c1dcb3e3 P 371e7f54fd044a1ea8a74b65877495f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:21.534866 23189 tablet_bootstrap.cc:492] T 8474ba477f0d49bb9b181b7336b66b7c P a236b1e31f374fdcae1e1ced5829b638: No bootstrap required, opened a new log
I20260928 14:53:21.534919 23189 ts_tablet_manager.cc:1397] T 8474ba477f0d49bb9b181b7336b66b7c P a236b1e31f374fdcae1e1ced5829b638: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:21.534900 23237 raft_consensus.cc:2258] T 5f67640289134f4485c343e5c1dcb3e3 P 371e7f54fd044a1ea8a74b65877495f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:21.535053 23189 raft_consensus.cc:348] T 8474ba477f0d49bb9b181b7336b66b7c P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:21.535120 23189 raft_consensus.cc:374] T 8474ba477f0d49bb9b181b7336b66b7c P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:21.535140 23189 raft_consensus.cc:729] T 8474ba477f0d49bb9b181b7336b66b7c P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a236b1e31f374fdcae1e1ced5829b638, State: Initialized, Role: FOLLOWER
I20260928 14:53:21.535182 23189 consensus_queue.cc:260] T 8474ba477f0d49bb9b181b7336b66b7c P a236b1e31f374fdcae1e1ced5829b638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:21.535274 23189 ts_tablet_manager.cc:1428] T 8474ba477f0d49bb9b181b7336b66b7c P a236b1e31f374fdcae1e1ced5829b638: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:21.535548 23237 ts_tablet_manager.cc:1905] T 5f67640289134f4485c343e5c1dcb3e3 P 371e7f54fd044a1ea8a74b65877495f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:21.537475 23237 ts_tablet_manager.cc:1918] T 5f67640289134f4485c343e5c1dcb3e3 P 371e7f54fd044a1ea8a74b65877495f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:53:21.537590 23237 log.cc:1196] T 5f67640289134f4485c343e5c1dcb3e3 P 371e7f54fd044a1ea8a74b65877495f8: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/5f67640289134f4485c343e5c1dcb3e3
I20260928 14:53:21.538197 23237 ts_tablet_manager.cc:1939] T 5f67640289134f4485c343e5c1dcb3e3 P 371e7f54fd044a1ea8a74b65877495f8: Deleting consensus metadata
I20260928 14:53:21.538646 23238 tablet_replica.cc:330] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f: stopping tablet replica
I20260928 14:53:21.538686 22663 catalog_manager.cc:4841] TS 371e7f54fd044a1ea8a74b65877495f8 (127.20.116.130:40537): tablet 5f67640289134f4485c343e5c1dcb3e3 (table test-table [id=c7757d0e6afa47b8a326ecf4160ca247]) successfully deleted
I20260928 14:53:21.538841 23238 raft_consensus.cc:2229] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:53:21.538993 23238 raft_consensus.cc:2258] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:21.539392 23238 ts_tablet_manager.cc:1905] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:21.541173 23238 ts_tablet_manager.cc:1918] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:53:21.541296 23238 log.cc:1196] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/5f67640289134f4485c343e5c1dcb3e3
I20260928 14:53:21.541608 23238 ts_tablet_manager.cc:1939] T 5f67640289134f4485c343e5c1dcb3e3 P edc88735caac449ab12811d019c3061f: Deleting consensus metadata
I20260928 14:53:21.542052 22663 catalog_manager.cc:4841] TS edc88735caac449ab12811d019c3061f (127.20.116.131:38345): tablet 5f67640289134f4485c343e5c1dcb3e3 (table test-table [id=c7757d0e6afa47b8a326ecf4160ca247]) successfully deleted
I20260928 14:53:21.635416 23193 raft_consensus.cc:482] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:53:21.635556 23193 raft_consensus.cc:504] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:21.635735 23193 leader_election.cc:288] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a236b1e31f374fdcae1e1ced5829b638 (127.20.116.129:32831), 371e7f54fd044a1ea8a74b65877495f8 (127.20.116.130:40537)
I20260928 14:53:21.636014 22835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8474ba477f0d49bb9b181b7336b66b7c" candidate_uuid: "edc88735caac449ab12811d019c3061f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a236b1e31f374fdcae1e1ced5829b638" is_pre_election: true
I20260928 14:53:21.636041 22976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8474ba477f0d49bb9b181b7336b66b7c" candidate_uuid: "edc88735caac449ab12811d019c3061f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371e7f54fd044a1ea8a74b65877495f8" is_pre_election: true
I20260928 14:53:21.636134 22835 raft_consensus.cc:2454] T 8474ba477f0d49bb9b181b7336b66b7c P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc88735caac449ab12811d019c3061f in term 0.
I20260928 14:53:21.636133 22976 raft_consensus.cc:2454] T 8474ba477f0d49bb9b181b7336b66b7c P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc88735caac449ab12811d019c3061f in term 0.
I20260928 14:53:21.636315 23051 leader_election.cc:302] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a236b1e31f374fdcae1e1ced5829b638, edc88735caac449ab12811d019c3061f; no voters:
I20260928 14:53:21.636441 23193 raft_consensus.cc:2781] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:53:21.636495 23193 raft_consensus.cc:482] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:53:21.636516 23193 raft_consensus.cc:3037] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:21.637161 23193 raft_consensus.cc:504] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:21.637302 23193 leader_election.cc:288] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f [CANDIDATE]: Term 1 election: Requested vote from peers a236b1e31f374fdcae1e1ced5829b638 (127.20.116.129:32831), 371e7f54fd044a1ea8a74b65877495f8 (127.20.116.130:40537)
I20260928 14:53:21.637485 22976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8474ba477f0d49bb9b181b7336b66b7c" candidate_uuid: "edc88735caac449ab12811d019c3061f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371e7f54fd044a1ea8a74b65877495f8"
I20260928 14:53:21.637496 22835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8474ba477f0d49bb9b181b7336b66b7c" candidate_uuid: "edc88735caac449ab12811d019c3061f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a236b1e31f374fdcae1e1ced5829b638"
I20260928 14:53:21.637583 22976 raft_consensus.cc:3037] T 8474ba477f0d49bb9b181b7336b66b7c P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:21.637591 22835 raft_consensus.cc:3037] T 8474ba477f0d49bb9b181b7336b66b7c P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:21.638167 22976 raft_consensus.cc:2454] T 8474ba477f0d49bb9b181b7336b66b7c P 371e7f54fd044a1ea8a74b65877495f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc88735caac449ab12811d019c3061f in term 1.
I20260928 14:53:21.638335 23051 leader_election.cc:302] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371e7f54fd044a1ea8a74b65877495f8, edc88735caac449ab12811d019c3061f; no voters:
I20260928 14:53:21.638437 23193 raft_consensus.cc:2781] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:21.638490 23193 raft_consensus.cc:686] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f [term 1 LEADER]: Becoming Leader. State: Replica: edc88735caac449ab12811d019c3061f, State: Running, Role: LEADER
I20260928 14:53:21.638571 23193 consensus_queue.cc:237] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:21.638620 22835 raft_consensus.cc:2454] T 8474ba477f0d49bb9b181b7336b66b7c P a236b1e31f374fdcae1e1ced5829b638 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc88735caac449ab12811d019c3061f in term 1.
I20260928 14:53:21.639161 22675 catalog_manager.cc:5495] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f reported cstate change: term changed from 0 to 1, leader changed from <none> to edc88735caac449ab12811d019c3061f (127.20.116.131). New cstate: current_term: 1 leader_uuid: "edc88735caac449ab12811d019c3061f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } health_report { overall_health: HEALTHY } } }
I20260928 14:53:21.656251 22835 raft_consensus.cc:1261] T 8474ba477f0d49bb9b181b7336b66b7c P a236b1e31f374fdcae1e1ced5829b638 [term 1 FOLLOWER]: Refusing update from remote peer edc88735caac449ab12811d019c3061f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:21.656240 22976 raft_consensus.cc:1261] T 8474ba477f0d49bb9b181b7336b66b7c P 371e7f54fd044a1ea8a74b65877495f8 [term 1 FOLLOWER]: Refusing update from remote peer edc88735caac449ab12811d019c3061f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:21.656498 23193 consensus_queue.cc:1035] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:21.656649 23197 consensus_queue.cc:1035] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f [LEADER]: Connected to new peer: Peer: permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 14:53:21.660980 23217 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 14:53:23.179699 22675 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:23.179811 22675 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:23.180572 22675 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7: Sending DeleteTablet for 3 replicas of tablet 8474ba477f0d49bb9b181b7336b66b7c
I20260928 14:53:23.180892 22956 tablet_service.cc:1514] Processing DeleteTablet for tablet 8474ba477f0d49bb9b181b7336b66b7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:23 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:39336
I20260928 14:53:23.180933 22815 tablet_service.cc:1514] Processing DeleteTablet for tablet 8474ba477f0d49bb9b181b7336b66b7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:23 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:57046
I20260928 14:53:23.180928 23097 tablet_service.cc:1514] Processing DeleteTablet for tablet 8474ba477f0d49bb9b181b7336b66b7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:23 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:34024
I20260928 14:53:23.181137 23267 tablet_replica.cc:330] T 8474ba477f0d49bb9b181b7336b66b7c P 371e7f54fd044a1ea8a74b65877495f8: stopping tablet replica
I20260928 14:53:23.181201 23267 raft_consensus.cc:2229] T 8474ba477f0d49bb9b181b7336b66b7c P 371e7f54fd044a1ea8a74b65877495f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:23.181373 23269 tablet_replica.cc:330] T 8474ba477f0d49bb9b181b7336b66b7c P a236b1e31f374fdcae1e1ced5829b638: stopping tablet replica
I20260928 14:53:23.181440 23269 raft_consensus.cc:2229] T 8474ba477f0d49bb9b181b7336b66b7c P a236b1e31f374fdcae1e1ced5829b638 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:23.181484 23269 raft_consensus.cc:2258] T 8474ba477f0d49bb9b181b7336b66b7c P a236b1e31f374fdcae1e1ced5829b638 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:23.181237 23267 raft_consensus.cc:2258] T 8474ba477f0d49bb9b181b7336b66b7c P 371e7f54fd044a1ea8a74b65877495f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:23.181831 23269 ts_tablet_manager.cc:1905] T 8474ba477f0d49bb9b181b7336b66b7c P a236b1e31f374fdcae1e1ced5829b638: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:23.183935 23267 ts_tablet_manager.cc:1905] T 8474ba477f0d49bb9b181b7336b66b7c P 371e7f54fd044a1ea8a74b65877495f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:23.183943 23269 ts_tablet_manager.cc:1918] T 8474ba477f0d49bb9b181b7336b66b7c P a236b1e31f374fdcae1e1ced5829b638: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:53:23.184000 23269 log.cc:1196] T 8474ba477f0d49bb9b181b7336b66b7c P a236b1e31f374fdcae1e1ced5829b638: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-0/wal/wals/8474ba477f0d49bb9b181b7336b66b7c
I20260928 14:53:23.184332 23269 ts_tablet_manager.cc:1939] T 8474ba477f0d49bb9b181b7336b66b7c P a236b1e31f374fdcae1e1ced5829b638: Deleting consensus metadata
I20260928 14:53:23.184686 22663 catalog_manager.cc:4841] TS a236b1e31f374fdcae1e1ced5829b638 (127.20.116.129:32831): tablet 8474ba477f0d49bb9b181b7336b66b7c (table test-table [id=b992c2e4355f4d64b26ba6bcb763c69f]) successfully deleted
I20260928 14:53:23.185421 23267 ts_tablet_manager.cc:1918] T 8474ba477f0d49bb9b181b7336b66b7c P 371e7f54fd044a1ea8a74b65877495f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:53:23.185508 23267 log.cc:1196] T 8474ba477f0d49bb9b181b7336b66b7c P 371e7f54fd044a1ea8a74b65877495f8: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/8474ba477f0d49bb9b181b7336b66b7c
I20260928 14:53:23.185714 23267 ts_tablet_manager.cc:1939] T 8474ba477f0d49bb9b181b7336b66b7c P 371e7f54fd044a1ea8a74b65877495f8: Deleting consensus metadata
I20260928 14:53:23.186091 22663 catalog_manager.cc:4841] TS 371e7f54fd044a1ea8a74b65877495f8 (127.20.116.130:40537): tablet 8474ba477f0d49bb9b181b7336b66b7c (table test-table [id=b992c2e4355f4d64b26ba6bcb763c69f]) successfully deleted
I20260928 14:53:23.186630 23268 tablet_replica.cc:330] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f: stopping tablet replica
I20260928 14:53:23.186707 23268 raft_consensus.cc:2229] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:53:23.186793 23268 raft_consensus.cc:2258] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:23.187057 23268 ts_tablet_manager.cc:1905] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:23.188387 23268 ts_tablet_manager.cc:1918] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:53:23.188457 23268 log.cc:1196] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/8474ba477f0d49bb9b181b7336b66b7c
I20260928 14:53:23.188773 23268 ts_tablet_manager.cc:1939] T 8474ba477f0d49bb9b181b7336b66b7c P edc88735caac449ab12811d019c3061f: Deleting consensus metadata
I20260928 14:53:23.189112 22663 catalog_manager.cc:4841] TS edc88735caac449ab12811d019c3061f (127.20.116.131:38345): tablet 8474ba477f0d49bb9b181b7336b66b7c (table test-table [id=b992c2e4355f4d64b26ba6bcb763c69f]) successfully deleted
Sep 28 14:53:23 dist-test-slave-pxl6 krb5kdc[22631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607203, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:53:23 dist-test-slave-pxl6 krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607203, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:23.212388 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:23.199005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54082 (local address 127.20.116.190:39027)
0928 14:53:23.199175 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:23.199176 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:23.199185 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:53:23.199207 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:23.199208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:23.199219 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:23.199278 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:23.199864 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.200359 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.201130 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.201319 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.201368 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:23.209550 (+ 8182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:23.209552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:23.209552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:23.209560 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:23.211238 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:23.211601 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:23.211602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:23.211602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:23.211655 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:23.211891 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:23.211892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:23.211892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:23.212030 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:23.212123 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:23.212220 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 14:53:23.212292 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":114,"threads_started":1}
I20260928 14:53:23.214118 22675 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:53:23.214303 22675 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:53:23.215878 22956 tablet_service.cc:1467] Processing CreateTablet for tablet 45514ae100bc4674902232333a8d8337 (DEFAULT_TABLE table=test-table [id=72f9b15f76964588a3cd9cbe88ea2534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:23.215876 22815 tablet_service.cc:1467] Processing CreateTablet for tablet 45514ae100bc4674902232333a8d8337 (DEFAULT_TABLE table=test-table [id=72f9b15f76964588a3cd9cbe88ea2534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:23.215900 23097 tablet_service.cc:1467] Processing CreateTablet for tablet 45514ae100bc4674902232333a8d8337 (DEFAULT_TABLE table=test-table [id=72f9b15f76964588a3cd9cbe88ea2534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:23.216018 23097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45514ae100bc4674902232333a8d8337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:23.216019 22815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45514ae100bc4674902232333a8d8337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:23.216043 22956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45514ae100bc4674902232333a8d8337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:23.217443 23279 tablet_bootstrap.cc:492] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638: Bootstrap starting.
I20260928 14:53:23.217443 23280 tablet_bootstrap.cc:492] T 45514ae100bc4674902232333a8d8337 P edc88735caac449ab12811d019c3061f: Bootstrap starting.
I20260928 14:53:23.217590 23278 tablet_bootstrap.cc:492] T 45514ae100bc4674902232333a8d8337 P 371e7f54fd044a1ea8a74b65877495f8: Bootstrap starting.
I20260928 14:53:23.217823 23279 tablet_bootstrap.cc:654] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:23.217821 23280 tablet_bootstrap.cc:654] T 45514ae100bc4674902232333a8d8337 P edc88735caac449ab12811d019c3061f: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:23.218081 23278 tablet_bootstrap.cc:654] T 45514ae100bc4674902232333a8d8337 P 371e7f54fd044a1ea8a74b65877495f8: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:23.218524 23280 tablet_bootstrap.cc:492] T 45514ae100bc4674902232333a8d8337 P edc88735caac449ab12811d019c3061f: No bootstrap required, opened a new log
I20260928 14:53:23.218616 23280 ts_tablet_manager.cc:1397] T 45514ae100bc4674902232333a8d8337 P edc88735caac449ab12811d019c3061f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:23.218704 23278 tablet_bootstrap.cc:492] T 45514ae100bc4674902232333a8d8337 P 371e7f54fd044a1ea8a74b65877495f8: No bootstrap required, opened a new log
I20260928 14:53:23.218722 23279 tablet_bootstrap.cc:492] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638: No bootstrap required, opened a new log
I20260928 14:53:23.218753 23278 ts_tablet_manager.cc:1397] T 45514ae100bc4674902232333a8d8337 P 371e7f54fd044a1ea8a74b65877495f8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:23.218771 23279 ts_tablet_manager.cc:1397] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:23.218745 23280 raft_consensus.cc:348] T 45514ae100bc4674902232333a8d8337 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:23.218823 23280 raft_consensus.cc:374] T 45514ae100bc4674902232333a8d8337 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:23.218891 23280 raft_consensus.cc:729] T 45514ae100bc4674902232333a8d8337 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc88735caac449ab12811d019c3061f, State: Initialized, Role: FOLLOWER
I20260928 14:53:23.218895 23278 raft_consensus.cc:348] T 45514ae100bc4674902232333a8d8337 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:23.218966 23278 raft_consensus.cc:374] T 45514ae100bc4674902232333a8d8337 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:23.218955 23280 consensus_queue.cc:260] T 45514ae100bc4674902232333a8d8337 P edc88735caac449ab12811d019c3061f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:23.218987 23278 raft_consensus.cc:729] T 45514ae100bc4674902232333a8d8337 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371e7f54fd044a1ea8a74b65877495f8, State: Initialized, Role: FOLLOWER
I20260928 14:53:23.218950 23279 raft_consensus.cc:348] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:23.219024 23279 raft_consensus.cc:374] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:23.219044 23278 consensus_queue.cc:260] T 45514ae100bc4674902232333a8d8337 P 371e7f54fd044a1ea8a74b65877495f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:23.219118 23279 raft_consensus.cc:729] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a236b1e31f374fdcae1e1ced5829b638, State: Initialized, Role: FOLLOWER
I20260928 14:53:23.219129 23280 ts_tablet_manager.cc:1428] T 45514ae100bc4674902232333a8d8337 P edc88735caac449ab12811d019c3061f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:23.219218 23278 ts_tablet_manager.cc:1428] T 45514ae100bc4674902232333a8d8337 P 371e7f54fd044a1ea8a74b65877495f8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:53:23.219211 23279 consensus_queue.cc:260] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:23.219358 23279 ts_tablet_manager.cc:1428] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:53:23.250661 23231 raft_consensus.cc:482] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:53:23.250787 23231 raft_consensus.cc:504] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:23.251019 23231 leader_election.cc:288] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371e7f54fd044a1ea8a74b65877495f8 (127.20.116.130:40537), edc88735caac449ab12811d019c3061f (127.20.116.131:38345)
I20260928 14:53:23.251294 23117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45514ae100bc4674902232333a8d8337" candidate_uuid: "a236b1e31f374fdcae1e1ced5829b638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc88735caac449ab12811d019c3061f" is_pre_election: true
I20260928 14:53:23.251385 22976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45514ae100bc4674902232333a8d8337" candidate_uuid: "a236b1e31f374fdcae1e1ced5829b638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371e7f54fd044a1ea8a74b65877495f8" is_pre_election: true
I20260928 14:53:23.251437 23117 raft_consensus.cc:2454] T 45514ae100bc4674902232333a8d8337 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a236b1e31f374fdcae1e1ced5829b638 in term 0.
I20260928 14:53:23.251472 22976 raft_consensus.cc:2454] T 45514ae100bc4674902232333a8d8337 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a236b1e31f374fdcae1e1ced5829b638 in term 0.
I20260928 14:53:23.251649 22768 leader_election.cc:302] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a236b1e31f374fdcae1e1ced5829b638, edc88735caac449ab12811d019c3061f; no voters:
I20260928 14:53:23.251777 23231 raft_consensus.cc:2781] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:53:23.251832 23231 raft_consensus.cc:482] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:53:23.251857 23231 raft_consensus.cc:3037] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:23.252998 23231 raft_consensus.cc:504] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:23.253149 23231 leader_election.cc:288] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638 [CANDIDATE]: Term 1 election: Requested vote from peers 371e7f54fd044a1ea8a74b65877495f8 (127.20.116.130:40537), edc88735caac449ab12811d019c3061f (127.20.116.131:38345)
I20260928 14:53:23.253438 22976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45514ae100bc4674902232333a8d8337" candidate_uuid: "a236b1e31f374fdcae1e1ced5829b638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371e7f54fd044a1ea8a74b65877495f8"
I20260928 14:53:23.253513 22976 raft_consensus.cc:3037] T 45514ae100bc4674902232333a8d8337 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:23.253502 23117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45514ae100bc4674902232333a8d8337" candidate_uuid: "a236b1e31f374fdcae1e1ced5829b638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc88735caac449ab12811d019c3061f"
I20260928 14:53:23.253609 23117 raft_consensus.cc:3037] T 45514ae100bc4674902232333a8d8337 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:23.254109 22976 raft_consensus.cc:2454] T 45514ae100bc4674902232333a8d8337 P 371e7f54fd044a1ea8a74b65877495f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a236b1e31f374fdcae1e1ced5829b638 in term 1.
I20260928 14:53:23.254266 23117 raft_consensus.cc:2454] T 45514ae100bc4674902232333a8d8337 P edc88735caac449ab12811d019c3061f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a236b1e31f374fdcae1e1ced5829b638 in term 1.
I20260928 14:53:23.254277 22768 leader_election.cc:302] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371e7f54fd044a1ea8a74b65877495f8, a236b1e31f374fdcae1e1ced5829b638; no voters:
I20260928 14:53:23.254479 23231 raft_consensus.cc:2781] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:23.254585 23231 raft_consensus.cc:686] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638 [term 1 LEADER]: Becoming Leader. State: Replica: a236b1e31f374fdcae1e1ced5829b638, State: Running, Role: LEADER
I20260928 14:53:23.254658 23231 consensus_queue.cc:237] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:23.255299 22674 catalog_manager.cc:5495] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638 reported cstate change: term changed from 0 to 1, leader changed from <none> to a236b1e31f374fdcae1e1ced5829b638 (127.20.116.129). New cstate: current_term: 1 leader_uuid: "a236b1e31f374fdcae1e1ced5829b638" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } health_report { overall_health: UNKNOWN } } }
I20260928 14:53:23.269394 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:23.266814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57072 (local address 127.20.116.129:32831)
0928 14:53:23.266958 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:23.266960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:23.266968 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:53:23.266998 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:23.266999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:23.267010 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:23.267074 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:23.267189 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.267723 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.268545 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.268785 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.268865 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:23.268996 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:23.269027 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:23.269070 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:23.269210 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 14:53:23.269234 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":94,"threads_started":1}
I20260928 14:53:23.274792 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:23.272243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:32831 (local address 127.20.116.129:58219)
0928 14:53:23.272418 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:23.272428 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:23.272499 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:23.272618 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:23.272619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:23.272620 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:53:23.272677 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:23.272678 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.273284 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.273285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:23.273864 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:23.273866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.274232 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.274233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:23.274282 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:23.274286 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:53:23.274515 (+ 229us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:53:23.274517 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:53:23.274589 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 14:53:23.274654 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":3,"thread_start_us":106,"threads_started":1}
I20260928 14:53:23.274792 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:23.272296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:58219 (local address 127.20.116.129:32831)
0928 14:53:23.272330 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:23.272331 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:23.272485 (+ 154us) server_negotiation.cc:407] Connection header received
0928 14:53:23.272557 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:23.272558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:23.272569 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:23.272618 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:23.272731 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.273200 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.273949 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.274148 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.274222 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:23.274374 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:23.274413 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:23.274507 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:23.274629 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 14:53:23.274649 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":17}
I20260928 14:53:23.277307 23117 raft_consensus.cc:1261] T 45514ae100bc4674902232333a8d8337 P edc88735caac449ab12811d019c3061f [term 1 FOLLOWER]: Refusing update from remote peer a236b1e31f374fdcae1e1ced5829b638: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:23.277318 22976 raft_consensus.cc:1261] T 45514ae100bc4674902232333a8d8337 P 371e7f54fd044a1ea8a74b65877495f8 [term 1 FOLLOWER]: Refusing update from remote peer a236b1e31f374fdcae1e1ced5829b638: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:23.277573 23248 consensus_queue.cc:1035] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:23.277695 23231 consensus_queue.cc:1035] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:23.288712 22675 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:23.288844 22675 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:23.289513 22675 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7: Sending DeleteTablet for 3 replicas of tablet 45514ae100bc4674902232333a8d8337
I20260928 14:53:23.289873 22814 tablet_service.cc:1514] Processing DeleteTablet for tablet 45514ae100bc4674902232333a8d8337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:23 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:57046
I20260928 14:53:23.289886 22956 tablet_service.cc:1514] Processing DeleteTablet for tablet 45514ae100bc4674902232333a8d8337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:23 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:39336
I20260928 14:53:23.289912 23097 tablet_service.cc:1514] Processing DeleteTablet for tablet 45514ae100bc4674902232333a8d8337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:23 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:34024
I20260928 14:53:23.289994 23267 tablet_replica.cc:330] T 45514ae100bc4674902232333a8d8337 P 371e7f54fd044a1ea8a74b65877495f8: stopping tablet replica
I20260928 14:53:23.290011 23269 tablet_replica.cc:330] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638: stopping tablet replica
I20260928 14:53:23.290011 23268 tablet_replica.cc:330] T 45514ae100bc4674902232333a8d8337 P edc88735caac449ab12811d019c3061f: stopping tablet replica
I20260928 14:53:23.290050 23267 raft_consensus.cc:2229] T 45514ae100bc4674902232333a8d8337 P 371e7f54fd044a1ea8a74b65877495f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:23.290068 23268 raft_consensus.cc:2229] T 45514ae100bc4674902232333a8d8337 P edc88735caac449ab12811d019c3061f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:23.290066 23269 raft_consensus.cc:2229] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:53:23.290096 23267 raft_consensus.cc:2258] T 45514ae100bc4674902232333a8d8337 P 371e7f54fd044a1ea8a74b65877495f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:23.290107 23268 raft_consensus.cc:2258] T 45514ae100bc4674902232333a8d8337 P edc88735caac449ab12811d019c3061f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:23.290179 23269 raft_consensus.cc:2258] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:23.290448 23268 ts_tablet_manager.cc:1905] T 45514ae100bc4674902232333a8d8337 P edc88735caac449ab12811d019c3061f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:23.290453 23269 ts_tablet_manager.cc:1905] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:23.290846 22675 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:53:23.290993 22675 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:53:23.292414 23268 ts_tablet_manager.cc:1918] T 45514ae100bc4674902232333a8d8337 P edc88735caac449ab12811d019c3061f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:53:23.292491 23268 log.cc:1196] T 45514ae100bc4674902232333a8d8337 P edc88735caac449ab12811d019c3061f: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/45514ae100bc4674902232333a8d8337
I20260928 14:53:23.292604 23097 tablet_service.cc:1467] Processing CreateTablet for tablet 106fd2a2483842419bebdef97af76ad8 (DEFAULT_TABLE table=test-table [id=bb8a944470c64abaaefcd8813302d157]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:23.292701 23097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106fd2a2483842419bebdef97af76ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:23.292730 23268 ts_tablet_manager.cc:1939] T 45514ae100bc4674902232333a8d8337 P edc88735caac449ab12811d019c3061f: Deleting consensus metadata
I20260928 14:53:23.293049 22663 catalog_manager.cc:4841] TS edc88735caac449ab12811d019c3061f (127.20.116.131:38345): tablet 45514ae100bc4674902232333a8d8337 (table test-table [id=72f9b15f76964588a3cd9cbe88ea2534]) successfully deleted
I20260928 14:53:23.293275 23267 ts_tablet_manager.cc:1905] T 45514ae100bc4674902232333a8d8337 P 371e7f54fd044a1ea8a74b65877495f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:23.293903 22956 tablet_service.cc:1467] Processing CreateTablet for tablet 106fd2a2483842419bebdef97af76ad8 (DEFAULT_TABLE table=test-table [id=bb8a944470c64abaaefcd8813302d157]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:23.294010 22956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106fd2a2483842419bebdef97af76ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:23.294654 22814 tablet_service.cc:1467] Processing CreateTablet for tablet 106fd2a2483842419bebdef97af76ad8 (DEFAULT_TABLE table=test-table [id=bb8a944470c64abaaefcd8813302d157]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:23.294703 23267 ts_tablet_manager.cc:1918] T 45514ae100bc4674902232333a8d8337 P 371e7f54fd044a1ea8a74b65877495f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:53:23.294752 23267 log.cc:1196] T 45514ae100bc4674902232333a8d8337 P 371e7f54fd044a1ea8a74b65877495f8: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/45514ae100bc4674902232333a8d8337
I20260928 14:53:23.294842 23269 ts_tablet_manager.cc:1918] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:53:23.294886 23269 log.cc:1196] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-0/wal/wals/45514ae100bc4674902232333a8d8337
I20260928 14:53:23.294966 23267 ts_tablet_manager.cc:1939] T 45514ae100bc4674902232333a8d8337 P 371e7f54fd044a1ea8a74b65877495f8: Deleting consensus metadata
I20260928 14:53:23.295106 23269 ts_tablet_manager.cc:1939] T 45514ae100bc4674902232333a8d8337 P a236b1e31f374fdcae1e1ced5829b638: Deleting consensus metadata
I20260928 14:53:23.295235 22814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106fd2a2483842419bebdef97af76ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:23.295239 23280 tablet_bootstrap.cc:492] T 106fd2a2483842419bebdef97af76ad8 P edc88735caac449ab12811d019c3061f: Bootstrap starting.
I20260928 14:53:23.295239 23278 tablet_bootstrap.cc:492] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8: Bootstrap starting.
I20260928 14:53:23.295553 23280 tablet_bootstrap.cc:654] T 106fd2a2483842419bebdef97af76ad8 P edc88735caac449ab12811d019c3061f: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:23.295842 23278 tablet_bootstrap.cc:654] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:23.295893 22663 catalog_manager.cc:4841] TS 371e7f54fd044a1ea8a74b65877495f8 (127.20.116.130:40537): tablet 45514ae100bc4674902232333a8d8337 (table test-table [id=72f9b15f76964588a3cd9cbe88ea2534]) successfully deleted
I20260928 14:53:23.296406 23280 tablet_bootstrap.cc:492] T 106fd2a2483842419bebdef97af76ad8 P edc88735caac449ab12811d019c3061f: No bootstrap required, opened a new log
I20260928 14:53:23.296406 23278 tablet_bootstrap.cc:492] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8: No bootstrap required, opened a new log
I20260928 14:53:23.296449 23278 ts_tablet_manager.cc:1397] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:23.296449 23280 ts_tablet_manager.cc:1397] T 106fd2a2483842419bebdef97af76ad8 P edc88735caac449ab12811d019c3061f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:23.296502 23279 tablet_bootstrap.cc:492] T 106fd2a2483842419bebdef97af76ad8 P a236b1e31f374fdcae1e1ced5829b638: Bootstrap starting.
I20260928 14:53:23.296607 23280 raft_consensus.cc:348] T 106fd2a2483842419bebdef97af76ad8 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:23.296623 23278 raft_consensus.cc:348] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:23.296674 23280 raft_consensus.cc:374] T 106fd2a2483842419bebdef97af76ad8 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:23.296681 23278 raft_consensus.cc:374] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:23.296710 23280 raft_consensus.cc:729] T 106fd2a2483842419bebdef97af76ad8 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc88735caac449ab12811d019c3061f, State: Initialized, Role: FOLLOWER
I20260928 14:53:23.296710 23278 raft_consensus.cc:729] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371e7f54fd044a1ea8a74b65877495f8, State: Initialized, Role: FOLLOWER
I20260928 14:53:23.296751 23280 consensus_queue.cc:260] T 106fd2a2483842419bebdef97af76ad8 P edc88735caac449ab12811d019c3061f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:23.296752 23278 consensus_queue.cc:260] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:23.296815 23279 tablet_bootstrap.cc:654] T 106fd2a2483842419bebdef97af76ad8 P a236b1e31f374fdcae1e1ced5829b638: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:23.296835 23278 ts_tablet_manager.cc:1428] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:23.296845 23280 ts_tablet_manager.cc:1428] T 106fd2a2483842419bebdef97af76ad8 P edc88735caac449ab12811d019c3061f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:23.297737 22663 catalog_manager.cc:4841] TS a236b1e31f374fdcae1e1ced5829b638 (127.20.116.129:32831): tablet 45514ae100bc4674902232333a8d8337 (table test-table [id=72f9b15f76964588a3cd9cbe88ea2534]) successfully deleted
I20260928 14:53:23.298283 23279 tablet_bootstrap.cc:492] T 106fd2a2483842419bebdef97af76ad8 P a236b1e31f374fdcae1e1ced5829b638: No bootstrap required, opened a new log
I20260928 14:53:23.298447 23279 ts_tablet_manager.cc:1397] T 106fd2a2483842419bebdef97af76ad8 P a236b1e31f374fdcae1e1ced5829b638: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:53:23.298641 23279 raft_consensus.cc:348] T 106fd2a2483842419bebdef97af76ad8 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:23.298754 23279 raft_consensus.cc:374] T 106fd2a2483842419bebdef97af76ad8 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:23.298780 23279 raft_consensus.cc:729] T 106fd2a2483842419bebdef97af76ad8 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a236b1e31f374fdcae1e1ced5829b638, State: Initialized, Role: FOLLOWER
I20260928 14:53:23.298831 23279 consensus_queue.cc:260] T 106fd2a2483842419bebdef97af76ad8 P a236b1e31f374fdcae1e1ced5829b638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:23.298941 23279 ts_tablet_manager.cc:1428] T 106fd2a2483842419bebdef97af76ad8 P a236b1e31f374fdcae1e1ced5829b638: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:23.400497 23284 raft_consensus.cc:482] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:53:23.400612 23284 raft_consensus.cc:504] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:23.401070 23284 leader_election.cc:288] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a236b1e31f374fdcae1e1ced5829b638 (127.20.116.129:32831), edc88735caac449ab12811d019c3061f (127.20.116.131:38345)
I20260928 14:53:23.404848 23299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:23.401680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:37741 (local address 127.20.116.131:38345)
0928 14:53:23.401852 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:23.401854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:23.401861 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:53:23.401887 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:23.401888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:23.401899 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:23.401992 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:23.402184 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.402793 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.403996 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.404441 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.404501 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:23.404509 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:23.404713 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 14:53:23.404767 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
I20260928 14:53:23.404868 23297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:23.401318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:32831 (local address 127.20.116.130:48201)
0928 14:53:23.401477 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:23.401491 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:23.401599 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:23.401851 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:23.401852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:23.401856 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:23.401944 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:23.401947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.403078 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.403079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:23.404025 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:23.404028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.404578 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.404579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:23.404681 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:23.404687 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:53:23.404746 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 14:53:23.404786 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260928 14:53:23.405154 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:23.401405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:48201 (local address 127.20.116.129:32831)
0928 14:53:23.401483 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:23.401486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:23.401583 (+ 97us) server_negotiation.cc:407] Connection header received
0928 14:53:23.401757 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:23.401759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:23.401771 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:23.401833 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:23.402027 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.402975 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.404110 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.404468 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.404564 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:23.404573 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:23.404771 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 14:53:23.404822 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:53:23.404870 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:23.401536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:38345 (local address 127.20.116.130:37741)
0928 14:53:23.401695 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:23.401706 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:23.401769 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:23.402012 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:23.402013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:23.402016 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:23.402089 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:23.402091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.402857 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.402859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:23.403898 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:23.403901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.404533 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.404534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:23.404620 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:23.404624 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:53:23.404691 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 14:53:23.404746 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":41,"thread_start_us":110,"threads_started":1}
I20260928 14:53:23.405447 22835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106fd2a2483842419bebdef97af76ad8" candidate_uuid: "371e7f54fd044a1ea8a74b65877495f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a236b1e31f374fdcae1e1ced5829b638" is_pre_election: true
I20260928 14:53:23.405475 23117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106fd2a2483842419bebdef97af76ad8" candidate_uuid: "371e7f54fd044a1ea8a74b65877495f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc88735caac449ab12811d019c3061f" is_pre_election: true
I20260928 14:53:23.405551 23117 raft_consensus.cc:2454] T 106fd2a2483842419bebdef97af76ad8 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371e7f54fd044a1ea8a74b65877495f8 in term 0.
I20260928 14:53:23.405551 22835 raft_consensus.cc:2454] T 106fd2a2483842419bebdef97af76ad8 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371e7f54fd044a1ea8a74b65877495f8 in term 0.
I20260928 14:53:23.405747 22910 leader_election.cc:302] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371e7f54fd044a1ea8a74b65877495f8, a236b1e31f374fdcae1e1ced5829b638; no voters:
I20260928 14:53:23.405913 23284 raft_consensus.cc:2781] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:53:23.405959 23284 raft_consensus.cc:482] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:53:23.405982 23284 raft_consensus.cc:3037] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:23.424602 23284 raft_consensus.cc:504] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:23.424844 23284 leader_election.cc:288] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8 [CANDIDATE]: Term 1 election: Requested vote from peers a236b1e31f374fdcae1e1ced5829b638 (127.20.116.129:32831), edc88735caac449ab12811d019c3061f (127.20.116.131:38345)
I20260928 14:53:23.425132 22835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106fd2a2483842419bebdef97af76ad8" candidate_uuid: "371e7f54fd044a1ea8a74b65877495f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a236b1e31f374fdcae1e1ced5829b638"
I20260928 14:53:23.425240 22835 raft_consensus.cc:3037] T 106fd2a2483842419bebdef97af76ad8 P a236b1e31f374fdcae1e1ced5829b638 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:23.425179 23117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106fd2a2483842419bebdef97af76ad8" candidate_uuid: "371e7f54fd044a1ea8a74b65877495f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc88735caac449ab12811d019c3061f"
I20260928 14:53:23.425309 23117 raft_consensus.cc:3037] T 106fd2a2483842419bebdef97af76ad8 P edc88735caac449ab12811d019c3061f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:23.425992 23117 raft_consensus.cc:2454] T 106fd2a2483842419bebdef97af76ad8 P edc88735caac449ab12811d019c3061f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371e7f54fd044a1ea8a74b65877495f8 in term 1.
I20260928 14:53:23.426218 22910 leader_election.cc:302] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371e7f54fd044a1ea8a74b65877495f8, edc88735caac449ab12811d019c3061f; no voters:
I20260928 14:53:23.426241 22835 raft_consensus.cc:2454] T 106fd2a2483842419bebdef97af76ad8 P a236b1e31f374fdcae1e1ced5829b638 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371e7f54fd044a1ea8a74b65877495f8 in term 1.
I20260928 14:53:23.426368 23284 raft_consensus.cc:2781] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:23.426569 23284 raft_consensus.cc:686] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8 [term 1 LEADER]: Becoming Leader. State: Replica: 371e7f54fd044a1ea8a74b65877495f8, State: Running, Role: LEADER
I20260928 14:53:23.426673 23284 consensus_queue.cc:237] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } }
I20260928 14:53:23.427534 22675 catalog_manager.cc:5495] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371e7f54fd044a1ea8a74b65877495f8 (127.20.116.130). New cstate: current_term: 1 leader_uuid: "371e7f54fd044a1ea8a74b65877495f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371e7f54fd044a1ea8a74b65877495f8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 } health_report { overall_health: UNKNOWN } } }
I20260928 14:53:23.447882 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:23.444795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39352 (local address 127.20.116.130:40537)
0928 14:53:23.445026 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:23.445027 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:23.445035 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:53:23.445072 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:23.445085 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:23.445106 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:23.445197 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:23.445322 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.445835 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.446695 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.446929 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.446995 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:23.447242 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:23.447369 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:23.447738 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:23.447759 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 14:53:23.447786 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":129,"threads_started":1}
I20260928 14:53:23.454052 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:23.451014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:56977 (local address 127.20.116.129:32831)
0928 14:53:23.451079 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:23.451082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:23.451175 (+ 93us) server_negotiation.cc:407] Connection header received
0928 14:53:23.451222 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:23.451222 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:23.451235 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:23.451283 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:23.451453 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.451986 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.453106 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.453350 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.453434 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:23.453724 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:23.453762 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:23.453821 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:23.453914 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 14:53:23.453941 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:53:23.454080 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:23.450850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:32831 (local address 127.20.116.130:56977)
0928 14:53:23.451106 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:23.451117 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:23.451174 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:23.451302 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:23.451303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:23.451305 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:53:23.451358 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:23.451359 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.452240 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.452241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:23.453028 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:23.453030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.453496 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.453497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:23.453566 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:23.453585 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:53:23.453841 (+ 256us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:53:23.453845 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:53:23.453893 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 14:53:23.453955 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":99,"threads_started":1}
I20260928 14:53:23.456377 22835 raft_consensus.cc:1261] T 106fd2a2483842419bebdef97af76ad8 P a236b1e31f374fdcae1e1ced5829b638 [term 1 FOLLOWER]: Refusing update from remote peer 371e7f54fd044a1ea8a74b65877495f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:23.456406 23117 raft_consensus.cc:1261] T 106fd2a2483842419bebdef97af76ad8 P edc88735caac449ab12811d019c3061f [term 1 FOLLOWER]: Refusing update from remote peer 371e7f54fd044a1ea8a74b65877495f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:23.456697 23300 consensus_queue.cc:1035] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc88735caac449ab12811d019c3061f" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:23.456840 23284 consensus_queue.cc:1035] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a236b1e31f374fdcae1e1ced5829b638" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 32831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:23.464354 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:23.462096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:48879 (local address 127.20.116.130:40537)
0928 14:53:23.462137 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:23.462139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:23.462148 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:53:23.462222 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:23.462223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:23.462233 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:23.462287 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:23.462410 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.462956 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.463770 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.463994 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.464060 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:23.464187 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:23.464213 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:23.464255 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:23.464288 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 14:53:23.464304 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:53:23.464375 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:23.462055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:40537 (local address 127.20.116.129:48879)
0928 14:53:23.462099 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:23.462111 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:23.462156 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:23.462290 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:23.462290 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:23.462292 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:53:23.462356 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:23.462357 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.463055 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.463056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:23.463687 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:23.463689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:23.464076 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:23.464077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:23.464131 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:23.464137 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:53:23.464255 (+ 118us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:53:23.464256 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 14:53:23.464286 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 14:53:23.464323 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
W20260928 14:53:23.464815 23249 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 14:53:24.989322 22675 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:24.989430 22675 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:24.990334 22675 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 719aa42e8b754f3d8de1fd8b77b052f7: Sending DeleteTablet for 3 replicas of tablet 106fd2a2483842419bebdef97af76ad8
I20260928 14:53:24.990651 22814 tablet_service.cc:1514] Processing DeleteTablet for tablet 106fd2a2483842419bebdef97af76ad8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:24 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:57046
I20260928 14:53:24.990629 22956 tablet_service.cc:1514] Processing DeleteTablet for tablet 106fd2a2483842419bebdef97af76ad8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:24 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:39336
I20260928 14:53:24.990661 23097 tablet_service.cc:1514] Processing DeleteTablet for tablet 106fd2a2483842419bebdef97af76ad8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:24 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:34024
I20260928 14:53:24.990832 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 22749
I20260928 14:53:24.991672 23334 tablet_replica.cc:330] T 106fd2a2483842419bebdef97af76ad8 P edc88735caac449ab12811d019c3061f: stopping tablet replica
I20260928 14:53:24.991756 23334 raft_consensus.cc:2229] T 106fd2a2483842419bebdef97af76ad8 P edc88735caac449ab12811d019c3061f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:24.991856 23334 raft_consensus.cc:2258] T 106fd2a2483842419bebdef97af76ad8 P edc88735caac449ab12811d019c3061f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:24.992174 23334 ts_tablet_manager.cc:1905] T 106fd2a2483842419bebdef97af76ad8 P edc88735caac449ab12811d019c3061f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:24.993255 23333 tablet_replica.cc:330] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8: stopping tablet replica
I20260928 14:53:24.993335 23333 raft_consensus.cc:2229] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:53:24.993440 23333 raft_consensus.cc:2258] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:24.993702 23333 ts_tablet_manager.cc:1905] T 106fd2a2483842419bebdef97af76ad8 P 371e7f54fd044a1ea8a74b65877495f8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 14:53:24.999219 22663 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.116.129:32831 (error 108)
I20260928 14:53:24.999431 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 22891
I20260928 14:53:25.000232 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:24.999729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:32831 (local address 127.0.0.1:57076)
0928 14:53:25.000142 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:25.000191 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.129:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":61,"threads_started":1}
W20260928 14:53:25.000401 22663 catalog_manager.cc:4584] TS a236b1e31f374fdcae1e1ced5829b638 (127.20.116.129:32831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 106fd2a2483842419bebdef97af76ad8: Network error: Client connection negotiation failed: client connection to 127.20.116.129:32831: connect: Connection refused (error 111)
I20260928 14:53:25.053421 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:25.053234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:32831 (local address 127.0.0.1:57080)
0928 14:53:25.053298 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:25.053375 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.129:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 14:53:25.053828 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:25.053736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:32831 (local address 127.0.0.1:57088)
0928 14:53:25.053780 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:25.053807 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.129:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 14:53:25.073300 23334 ts_tablet_manager.cc:1918] T 106fd2a2483842419bebdef97af76ad8 P edc88735caac449ab12811d019c3061f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:53:25.073407 23334 log.cc:1196] T 106fd2a2483842419bebdef97af76ad8 P edc88735caac449ab12811d019c3061f: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/106fd2a2483842419bebdef97af76ad8
I20260928 14:53:25.073661 23334 ts_tablet_manager.cc:1939] T 106fd2a2483842419bebdef97af76ad8 P edc88735caac449ab12811d019c3061f: Deleting consensus metadata
I20260928 14:53:25.078603 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 23032
I20260928 14:53:25.078693 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:25.078563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:40537 (local address 127.0.0.1:39368)
0928 14:53:25.078624 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:25.078655 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.130:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 14:53:25.098147 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 22644
I20260928 14:53:25.098191 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:25.098024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:38345 (local address 127.0.0.1:34050)
0928 14:53:25.098093 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:25.098133 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.131:38345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
2026-09-28T14:53:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6031 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:53:26 dist-test-slave-pxl6 krb5kdc[23339](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:53:26 dist-test-slave-pxl6 krb5kdc[23339](info): set up 2 sockets
Sep 28 14:53:26 dist-test-slave-pxl6 krb5kdc[23339](info): commencing operation
krb5kdc: starting...
W20260928 14:53:27.816663 20946 mini_kdc.cc:121] Time spent starting KDC: real 2.696s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:53:27 dist-test-slave-pxl6 krb5kdc[23339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:53:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:53:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:53:27.945634 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:37015
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:37015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:28.034888 23352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:28.035145 23352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:28.035183 23352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:28.036373 23352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:53:28.036444 23352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:28.036469 23352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:28.036521 23352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:53:28.036544 23352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:53:28.038074 23352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:37015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:37015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:28.038372 23352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:28.038635 23352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:28.043704 23357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:28.043761 23358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:28.043753 23352 server_base.cc:1034] running on GCE node
W20260928 14:53:28.043712 23360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:28.044178 23352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:28.044484 23352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:28.045645 23352 hybrid_clock.cc:648] HybridClock initialized: now 1790607208045626 us; error 38 us; skew 500 ppm
Sep 28 14:53:28 dist-test-slave-pxl6 krb5kdc[23339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607208, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:28.048300 23352 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:53:28.048856 23352 webserver.cc:460] Webserver started at http://127.20.116.190:38487/ using document root <none> and password file <none>
I20260928 14:53:28.049100 23352 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:28.049180 23352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:28.049337 23352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:28.050806 23352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "c44b90f3fa7949aaa9a667104af1b5d0"
format_stamp: "Formatted at 2026-09-28 14:53:28 on dist-test-slave-pxl6"
server_key: "cae7331ae0cda34142a9396c2c87ec61"
server_key_iv: "4e6eda030479aba42215ea7547655083"
server_key_version: "encryptionkey@0"
I20260928 14:53:28.051306 23352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "c44b90f3fa7949aaa9a667104af1b5d0"
format_stamp: "Formatted at 2026-09-28 14:53:28 on dist-test-slave-pxl6"
server_key: "cae7331ae0cda34142a9396c2c87ec61"
server_key_iv: "4e6eda030479aba42215ea7547655083"
server_key_version: "encryptionkey@0"
I20260928 14:53:28.053552 23352 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:53:28.054461 23367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:28.054692 23352 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:53:28.054805 23352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "c44b90f3fa7949aaa9a667104af1b5d0"
format_stamp: "Formatted at 2026-09-28 14:53:28 on dist-test-slave-pxl6"
server_key: "cae7331ae0cda34142a9396c2c87ec61"
server_key_iv: "4e6eda030479aba42215ea7547655083"
server_key_version: "encryptionkey@0"
I20260928 14:53:28.054899 23352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:28.073046 23352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:28.075244 23352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:28.075428 23352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:28.079928 23352 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:37015
I20260928 14:53:28.079972 23419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:37015 every 8 connection(s)
I20260928 14:53:28.080361 23352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:53:28.080972 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 23352
I20260928 14:53:28.081077 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:53:28.081277 20946 external_mini_cluster.cc:1428] Setting key e0cd1930cae7896b6883134606adc64b
I20260928 14:53:28.081570 23420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:28.085294 23420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0: Bootstrap starting.
I20260928 14:53:28.086086 23420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:28.086402 23420 log.cc:824] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:28.087194 23420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0: No bootstrap required, opened a new log
Sep 28 14:53:28 dist-test-slave-pxl6 krb5kdc[23339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:28.089668 23420 raft_consensus.cc:348] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44b90f3fa7949aaa9a667104af1b5d0" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 37015 } }
I20260928 14:53:28.089825 23420 raft_consensus.cc:374] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:28.089848 23420 raft_consensus.cc:729] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c44b90f3fa7949aaa9a667104af1b5d0, State: Initialized, Role: FOLLOWER
I20260928 14:53:28.090030 23420 consensus_queue.cc:260] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44b90f3fa7949aaa9a667104af1b5d0" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 37015 } }
I20260928 14:53:28.090119 23420 raft_consensus.cc:388] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:53:28.090165 23420 raft_consensus.cc:482] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:53:28.090245 23420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:28.090917 23420 raft_consensus.cc:504] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44b90f3fa7949aaa9a667104af1b5d0" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 37015 } }
I20260928 14:53:28.091042 23420 leader_election.cc:302] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c44b90f3fa7949aaa9a667104af1b5d0; no voters:
I20260928 14:53:28.091259 23420 leader_election.cc:288] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:53:28.091295 23425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:28.091408 23425 raft_consensus.cc:686] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0 [term 1 LEADER]: Becoming Leader. State: Replica: c44b90f3fa7949aaa9a667104af1b5d0, State: Running, Role: LEADER
I20260928 14:53:28.091559 23425 consensus_queue.cc:237] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44b90f3fa7949aaa9a667104af1b5d0" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 37015 } }
I20260928 14:53:28.091568 23420 sys_catalog.cc:564] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:28.092971 23426 sys_catalog.cc:455] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c44b90f3fa7949aaa9a667104af1b5d0. Latest consensus state: current_term: 1 leader_uuid: "c44b90f3fa7949aaa9a667104af1b5d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44b90f3fa7949aaa9a667104af1b5d0" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 37015 } } }
I20260928 14:53:28.093106 23426 sys_catalog.cc:458] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:28.093327 23427 sys_catalog.cc:455] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c44b90f3fa7949aaa9a667104af1b5d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44b90f3fa7949aaa9a667104af1b5d0" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 37015 } } }
I20260928 14:53:28.093425 23427 sys_catalog.cc:458] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:28.093583 23439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:53:28.093760 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:28.081831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49598 (local address 127.20.116.190:37015)
0928 14:53:28.082033 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:28.082041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:28.082068 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:53:28.082852 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:28.082861 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:28.082919 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:28.083041 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:28.083897 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:28.084899 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:28.085967 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:28.086433 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:28.086601 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:28.089459 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:28.089463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:28.089463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:28.089477 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:28.091706 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:28.092151 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:28.092154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:28.092155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:28.092269 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:28.092554 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:28.092554 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:28.092555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:28.092886 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:28.093041 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:28.093293 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 14:53:28.093434 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":106,"threads_started":1}
I20260928 14:53:28.094261 23439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:53:28.096063 23439 catalog_manager.cc:1347] Generated new cluster ID: 1d8711cb1f6040728f72a089b266f6e7
I20260928 14:53:28.096120 23439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:53:28.101763 23439 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:53:28.102337 23439 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:53:28.107016 23439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c44b90f3fa7949aaa9a667104af1b5d0: Generated new TSK 0
I20260928 14:53:28.107246 23439 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
I20260928 14:53:28.141636 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:37015
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:28.228765 23446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:28.228978 23446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:28.229004 23446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:28.230144 23446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:28.230176 23446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:28.230201 23446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:53:28.231627 23446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:37015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:28.231832 23446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:28.232012 23446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:28.237332 23452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:28.237349 23451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:28.237346 23454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:28.237550 23446 server_base.cc:1034] running on GCE node
I20260928 14:53:28.237771 23446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:28.238049 23446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:28.239226 23446 hybrid_clock.cc:648] HybridClock initialized: now 1790607208239222 us; error 39 us; skew 500 ppm
Sep 28 14:53:28 dist-test-slave-pxl6 krb5kdc[23339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607208, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:28.247819 23446 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:53:28.248347 23446 webserver.cc:460] Webserver started at http://127.20.116.129:41837/ using document root <none> and password file <none>
I20260928 14:53:28.248618 23446 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:28.248692 23446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:28.248842 23446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:28.249969 23446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "447d8fad73ee4e96a342a0c81cb86494"
format_stamp: "Formatted at 2026-09-28 14:53:28 on dist-test-slave-pxl6"
server_key: "dbbfb415e9754b07431abf11f4d051c0"
server_key_iv: "880b2097f53bf17656fe8c75452cd156"
server_key_version: "encryptionkey@0"
I20260928 14:53:28.250350 23446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "447d8fad73ee4e96a342a0c81cb86494"
format_stamp: "Formatted at 2026-09-28 14:53:28 on dist-test-slave-pxl6"
server_key: "dbbfb415e9754b07431abf11f4d051c0"
server_key_iv: "880b2097f53bf17656fe8c75452cd156"
server_key_version: "encryptionkey@0"
I20260928 14:53:28.251832 23446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:53:28.252609 23461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:28.252821 23446 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:53:28.252920 23446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "447d8fad73ee4e96a342a0c81cb86494"
format_stamp: "Formatted at 2026-09-28 14:53:28 on dist-test-slave-pxl6"
server_key: "dbbfb415e9754b07431abf11f4d051c0"
server_key_iv: "880b2097f53bf17656fe8c75452cd156"
server_key_version: "encryptionkey@0"
I20260928 14:53:28.253005 23446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:28.273247 23446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:28.275278 23446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:28.275447 23446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:28.275709 23446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:28.276028 23446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:28.276082 23446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:28.276116 23446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:28.276166 23446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:28.282150 23446 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:37787
I20260928 14:53:28.282194 23576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:37787 every 8 connection(s)
I20260928 14:53:28.282522 23446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:53:28.287293 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 23446
I20260928 14:53:28.287393 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:53:28.287572 20946 external_mini_cluster.cc:1428] Setting key f1959e3fc35f612d6930953bdefa7bea
Sep 28 14:53:28 dist-test-slave-pxl6 krb5kdc[23339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607208, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:28.292297 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:28.283353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:51585 (local address 127.20.116.190:37015)
0928 14:53:28.283401 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:28.283403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:28.283622 (+ 219us) server_negotiation.cc:407] Connection header received
0928 14:53:28.284117 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:28.284118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:28.284128 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:28.284189 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:28.285111 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:28.285623 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:28.286709 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:28.286911 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:28.286966 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:28.289245 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:28.289247 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:28.289247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:28.289256 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:28.290793 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:28.291226 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:28.291228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:28.291228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:28.291288 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:28.291579 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:28.291580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:28.291580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:28.291713 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:28.291794 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:28.292112 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 14:53:28.292200 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:53:28.292516 23579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:28.283318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:37015 (local address 127.20.116.129:51585)
0928 14:53:28.283486 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:28.283530 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:28.284029 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:28.284246 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:28.284250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:28.284820 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:28.285021 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:28.285025 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:28.285729 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:28.285730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:28.286616 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:28.286618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:28.286987 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:28.286987 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:28.287089 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:28.287730 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:28.287738 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:28.289136 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:28.290890 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:28.290898 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:28.290900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:28.291133 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:28.291360 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:28.291361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:28.291361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:28.291524 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:28.291784 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:28.291790 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:28.291979 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 14:53:28.292090 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 14:53:28.292261 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260928 14:53:28.293020 23577 heartbeater.cc:347] Connected to a master server at 127.20.116.190:37015
I20260928 14:53:28.293135 23577 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:28.293326 23577 heartbeater.cc:511] Master 127.20.116.190:37015 requested a full tablet report, sending...
I20260928 14:53:28.293890 23384 ts_manager.cc:194] Registered new tserver with Master: 447d8fad73ee4e96a342a0c81cb86494 (127.20.116.129:37787)
I20260928 14:53:28.294519 23384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:51585
WARNING: no policy specified for kudu/127.20.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
I20260928 14:53:28.314464 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:37015
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:28.397166 23582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:28.397433 23582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:28.397490 23582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:28.398792 23582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:28.398867 23582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:28.398914 23582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:53:28.400491 23582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:37015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:28.400800 23582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:28.401068 23582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:28.406476 23588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:28.406476 23590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:28.406486 23587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:28.406692 23582 server_base.cc:1034] running on GCE node
I20260928 14:53:28.407004 23582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:28.407285 23582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:28.408450 23582 hybrid_clock.cc:648] HybridClock initialized: now 1790607208408445 us; error 53 us; skew 500 ppm
Sep 28 14:53:28 dist-test-slave-pxl6 krb5kdc[23339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607208, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:28.411058 23582 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:53:28.411509 23582 webserver.cc:460] Webserver started at http://127.20.116.130:37717/ using document root <none> and password file <none>
I20260928 14:53:28.411773 23582 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:28.411841 23582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:28.411993 23582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:28.413100 23582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "899bfab53fc0416b90ff4c8085552261"
format_stamp: "Formatted at 2026-09-28 14:53:28 on dist-test-slave-pxl6"
server_key: "3142053b2f783b84f04f1fda8377f862"
server_key_iv: "079da7c9c10718f770b66303e58f52af"
server_key_version: "encryptionkey@0"
I20260928 14:53:28.413496 23582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "899bfab53fc0416b90ff4c8085552261"
format_stamp: "Formatted at 2026-09-28 14:53:28 on dist-test-slave-pxl6"
server_key: "3142053b2f783b84f04f1fda8377f862"
server_key_iv: "079da7c9c10718f770b66303e58f52af"
server_key_version: "encryptionkey@0"
I20260928 14:53:28.415009 23582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:53:28.415860 23597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:28.416043 23582 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:53:28.416152 23582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "899bfab53fc0416b90ff4c8085552261"
format_stamp: "Formatted at 2026-09-28 14:53:28 on dist-test-slave-pxl6"
server_key: "3142053b2f783b84f04f1fda8377f862"
server_key_iv: "079da7c9c10718f770b66303e58f52af"
server_key_version: "encryptionkey@0"
I20260928 14:53:28.416242 23582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:28.426887 23582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:28.429255 23582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:28.429422 23582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:28.429672 23582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:28.429996 23582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:28.430049 23582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:28.430090 23582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:28.430119 23582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:28.436121 23582 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:43279
I20260928 14:53:28.436182 23712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:43279 every 8 connection(s)
I20260928 14:53:28.436551 23582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:53:28.439625 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 23582
I20260928 14:53:28.439702 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:53:28.439885 20946 external_mini_cluster.cc:1428] Setting key 1b682f11055211aeda6535f0a95dd248
Sep 28 14:53:28 dist-test-slave-pxl6 krb5kdc[23339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607208, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:28.446681 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:28.437274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:52469 (local address 127.20.116.190:37015)
0928 14:53:28.437318 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:28.437320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:28.437524 (+ 204us) server_negotiation.cc:407] Connection header received
0928 14:53:28.438068 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:28.438070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:28.438082 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:28.438121 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:28.439179 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:28.439712 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:28.440971 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:28.441191 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:28.441276 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:28.443739 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:28.443740 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:28.443741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:28.443748 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:28.445206 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:28.445637 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:28.445639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:28.445639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:28.445700 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:28.445963 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:28.445963 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:28.445964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:28.446085 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:28.446178 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:28.446472 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 14:53:28.446583 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 14:53:28.446900 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:28.437246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:37015 (local address 127.20.116.130:52469)
0928 14:53:28.437391 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:28.437434 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:28.437957 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:28.438204 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:28.438208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:28.438859 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:28.439057 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:28.439062 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:28.439796 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:28.439797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:28.440871 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:28.440874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:28.441288 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:28.441289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:28.441421 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:28.441992 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:28.442000 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:28.443585 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:28.445308 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:28.445315 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:28.445318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:28.445546 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:28.445771 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:28.445772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:28.445772 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:28.445891 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:28.446172 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:28.446180 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:28.446327 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 14:53:28.446453 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 14:53:28.446652 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
I20260928 14:53:28.447602 23713 heartbeater.cc:347] Connected to a master server at 127.20.116.190:37015
I20260928 14:53:28.447722 23713 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:28.447932 23713 heartbeater.cc:511] Master 127.20.116.190:37015 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.116.131@KRBTEST.COM; defaulting to no policy
I20260928 14:53:28.448357 23384 ts_manager.cc:194] Registered new tserver with Master: 899bfab53fc0416b90ff4c8085552261 (127.20.116.130:43279)
I20260928 14:53:28.448733 23384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:52469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
I20260928 14:53:28.466382 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:37015
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:28.550189 23718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:28.550418 23718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:28.550444 23718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:28.551774 23718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:28.551822 23718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:28.551851 23718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:53:28.553251 23718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:37015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:28.553450 23718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:28.553642 23718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:28.559324 23724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:28.559324 23723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:28.559559 23718 server_base.cc:1034] running on GCE node
W20260928 14:53:28.559372 23726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:28.559896 23718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:28.560178 23718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:28.561344 23718 hybrid_clock.cc:648] HybridClock initialized: now 1790607208561322 us; error 45 us; skew 500 ppm
Sep 28 14:53:28 dist-test-slave-pxl6 krb5kdc[23339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607208, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:28.564021 23718 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:53:28.564484 23718 webserver.cc:460] Webserver started at http://127.20.116.131:35255/ using document root <none> and password file <none>
I20260928 14:53:28.564718 23718 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:28.564787 23718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:28.564939 23718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:28.566004 23718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "1a8e47bccbaf431bb7bdf3b3b475e4ec"
format_stamp: "Formatted at 2026-09-28 14:53:28 on dist-test-slave-pxl6"
server_key: "a32bc794adf28d64cfface07d3f4a6cf"
server_key_iv: "43245845e6b63ae4f04c77f3b3fe4c51"
server_key_version: "encryptionkey@0"
I20260928 14:53:28.566386 23718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "1a8e47bccbaf431bb7bdf3b3b475e4ec"
format_stamp: "Formatted at 2026-09-28 14:53:28 on dist-test-slave-pxl6"
server_key: "a32bc794adf28d64cfface07d3f4a6cf"
server_key_iv: "43245845e6b63ae4f04c77f3b3fe4c51"
server_key_version: "encryptionkey@0"
I20260928 14:53:28.567773 23718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:53:28.568500 23733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:28.568673 23718 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:53:28.568869 23718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "1a8e47bccbaf431bb7bdf3b3b475e4ec"
format_stamp: "Formatted at 2026-09-28 14:53:28 on dist-test-slave-pxl6"
server_key: "a32bc794adf28d64cfface07d3f4a6cf"
server_key_iv: "43245845e6b63ae4f04c77f3b3fe4c51"
server_key_version: "encryptionkey@0"
I20260928 14:53:28.568986 23718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:28.593663 23718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:28.596522 23718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:28.596650 23718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:28.596858 23718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:28.597155 23718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:28.597189 23718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:28.597215 23718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:28.597229 23718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:28.603739 23718 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:45551
I20260928 14:53:28.603790 23847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:45551 every 8 connection(s)
I20260928 14:53:28.604161 23718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:53:28 dist-test-slave-pxl6 krb5kdc[23339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607208, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:28.611917 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 23718
I20260928 14:53:28.611995 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:53:28.612174 20946 external_mini_cluster.cc:1428] Setting key 8901edbe87d8a74ee5d0e42df9de8ce5
I20260928 14:53:28.614298 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:28.604922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:37007 (local address 127.20.116.190:37015)
0928 14:53:28.604979 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:28.604981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:28.605190 (+ 209us) server_negotiation.cc:407] Connection header received
0928 14:53:28.605743 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:28.605744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:28.605756 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:28.605823 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:28.606887 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:28.607522 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:28.608589 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:28.608801 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:28.608892 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:28.611300 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:28.611302 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:28.611302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:28.611310 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:28.612852 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:28.613370 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:28.613371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:28.613371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:28.613427 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:28.613725 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:28.613726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:28.613726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:28.613838 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:28.613890 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:28.614168 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 14:53:28.614232 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:53:28.614543 23850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:28.604890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:37015 (local address 127.20.116.131:37007)
0928 14:53:28.605066 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:28.605110 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:28.605619 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:28.605905 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:28.605908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:28.606519 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:28.606804 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:28.606810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:28.607649 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:28.607651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:28.608489 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:28.608493 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:28.608903 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:28.608904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:28.609048 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:28.609623 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:28.609632 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:28.611202 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:28.612942 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:28.612950 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:28.612954 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:28.613300 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:28.613509 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:28.613511 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:28.613511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:28.613657 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:28.613895 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:28.613899 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:28.614044 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 14:53:28.614133 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 14:53:28.614266 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260928 14:53:28.615078 23848 heartbeater.cc:347] Connected to a master server at 127.20.116.190:37015
I20260928 14:53:28.615207 23848 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:28.615408 23848 heartbeater.cc:511] Master 127.20.116.190:37015 requested a full tablet report, sending...
I20260928 14:53:28.615836 23384 ts_manager.cc:194] Registered new tserver with Master: 1a8e47bccbaf431bb7bdf3b3b475e4ec (127.20.116.131:45551)
I20260928 14:53:28.616231 23384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:37007
I20260928 14:53:28.624359 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 14:53:28.636270 23423 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:28.635119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58032 (local address 127.20.116.190:37015)
0928 14:53:28.635220 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:28.635224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:28.635287 (+ 63us) server_negotiation.cc:407] Connection header received
0928 14:53:28.635337 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:28.635338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:28.635348 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:28.635402 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:28.636183 (+ 781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58032: BlockingRecv error: recv got EOF from 127.0.0.1:58032 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 14:53:28.637055 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 23446
I20260928 14:53:28.641914 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 23582
I20260928 14:53:28.646831 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 23718
I20260928 14:53:28.651266 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 23352
2026-09-28T14:53:28Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3550 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:53:28 dist-test-slave-pxl6 krb5kdc[23860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:53:28 dist-test-slave-pxl6 krb5kdc[23860](info): set up 2 sockets
Sep 28 14:53:28 dist-test-slave-pxl6 krb5kdc[23860](info): commencing operation
krb5kdc: starting...
W20260928 14:53:29.699587 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:53:29 dist-test-slave-pxl6 krb5kdc[23860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:53:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:53:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:53:29.825459 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:33517
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:41423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:33517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:29.909891 23873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:29.910097 23873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:29.910120 23873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:29.911307 23873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:53:29.911350 23873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:29.911365 23873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:29.911378 23873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:53:29.911391 23873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:53:29.912910 23873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:41423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:33517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:33517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:29.913161 23873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:29.913408 23873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:29.918412 23878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:29.918427 23879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:29.918486 23881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:29.918709 23873 server_base.cc:1034] running on GCE node
I20260928 14:53:29.918933 23873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:29.919191 23873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:29.920375 23873 hybrid_clock.cc:648] HybridClock initialized: now 1790607209920363 us; error 30 us; skew 500 ppm
Sep 28 14:53:29 dist-test-slave-pxl6 krb5kdc[23860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607209, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:29.922860 23873 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:53:29.923350 23873 webserver.cc:460] Webserver started at http://127.20.116.190:33335/ using document root <none> and password file <none>
I20260928 14:53:29.923578 23873 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:29.923652 23873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:29.923805 23873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:29.924892 23873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "105b052b368b4d9189e5b0aba6021382"
format_stamp: "Formatted at 2026-09-28 14:53:29 on dist-test-slave-pxl6"
server_key: "1d8dbb732a0891a18e7f525a1ba3299c"
server_key_iv: "1279eab77c9ddc4be18ecb6de2a287fb"
server_key_version: "encryptionkey@0"
I20260928 14:53:29.925258 23873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "105b052b368b4d9189e5b0aba6021382"
format_stamp: "Formatted at 2026-09-28 14:53:29 on dist-test-slave-pxl6"
server_key: "1d8dbb732a0891a18e7f525a1ba3299c"
server_key_iv: "1279eab77c9ddc4be18ecb6de2a287fb"
server_key_version: "encryptionkey@0"
I20260928 14:53:29.926632 23873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 14:53:29.927410 23888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:29.927699 23873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:29.927809 23873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "105b052b368b4d9189e5b0aba6021382"
format_stamp: "Formatted at 2026-09-28 14:53:29 on dist-test-slave-pxl6"
server_key: "1d8dbb732a0891a18e7f525a1ba3299c"
server_key_iv: "1279eab77c9ddc4be18ecb6de2a287fb"
server_key_version: "encryptionkey@0"
I20260928 14:53:29.927881 23873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:29.935380 23873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:29.937312 23873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:29.937474 23873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:29.941823 23873 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:33517
I20260928 14:53:29.941874 23940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:33517 every 8 connection(s)
I20260928 14:53:29.942404 23873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:53:29.943184 23941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:29.945820 23941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382: Bootstrap starting.
I20260928 14:53:29.946411 23941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:29.946776 23941 log.cc:824] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:29.947461 23941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382: No bootstrap required, opened a new log
I20260928 14:53:29.949097 23941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105b052b368b4d9189e5b0aba6021382" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 33517 } }
I20260928 14:53:29.949240 23941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:29.949263 23941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105b052b368b4d9189e5b0aba6021382, State: Initialized, Role: FOLLOWER
I20260928 14:53:29.949414 23941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105b052b368b4d9189e5b0aba6021382" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 33517 } }
I20260928 14:53:29.949496 23941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:53:29.949519 23941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:53:29.949554 23941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:29.950449 23941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105b052b368b4d9189e5b0aba6021382" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 33517 } }
I20260928 14:53:29.950619 23941 leader_election.cc:302] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 105b052b368b4d9189e5b0aba6021382; no voters:
I20260928 14:53:29.950827 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 23873
I20260928 14:53:29.950845 23941 leader_election.cc:288] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:53:29.950923 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:53:29.950927 23944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:29.951083 23944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382 [term 1 LEADER]: Becoming Leader. State: Replica: 105b052b368b4d9189e5b0aba6021382, State: Running, Role: LEADER
I20260928 14:53:29.951133 20946 external_mini_cluster.cc:1428] Setting key 37a791590022bb8ba4557870318903b6
I20260928 14:53:29.951133 23941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:29.951179 23944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105b052b368b4d9189e5b0aba6021382" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 33517 } }
I20260928 14:53:29.951617 23945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "105b052b368b4d9189e5b0aba6021382" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105b052b368b4d9189e5b0aba6021382" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 33517 } } }
I20260928 14:53:29.951702 23945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382 [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:29.951941 23946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 105b052b368b4d9189e5b0aba6021382. Latest consensus state: current_term: 1 leader_uuid: "105b052b368b4d9189e5b0aba6021382" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105b052b368b4d9189e5b0aba6021382" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 33517 } } }
I20260928 14:53:29.952275 23946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382 [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:29.952193 23953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:53:29.952992 23953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:53:29.954754 23953 catalog_manager.cc:1347] Generated new cluster ID: bc722f427ba14957a72d991105a0e832
I20260928 14:53:29.954811 23953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:53:29 dist-test-slave-pxl6 krb5kdc[23860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:29.963907 23954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:29.951753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54144 (local address 127.20.116.190:33517)
0928 14:53:29.952232 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:29.952244 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:29.952283 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:53:29.952429 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:29.952436 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:29.952519 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:29.952781 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:29.953470 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:29.954835 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:29.955930 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:29.956644 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:29.956782 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:29.959839 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:29.959845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:29.959846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:29.959861 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:29.962039 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:29.962557 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:29.962562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:29.962562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:29.962688 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:29.962914 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:29.962915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:29.962916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:29.963242 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:29.963389 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:29.963513 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 14:53:29.963655 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":73,"threads_started":1}
I20260928 14:53:29.964854 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:53:29.981810 23953 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:53:29.982697 23954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:29.977730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54158 (local address 127.20.116.190:33517)
0928 14:53:29.977772 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:29.977774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:29.977939 (+ 165us) server_negotiation.cc:407] Connection header received
0928 14:53:29.977998 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:29.977999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:29.978012 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:29.978078 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:29.978500 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:29.979664 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:29.980682 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:29.981001 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:29.981082 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:29.982058 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:29.982060 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:29.982060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:53:29.982296 (+ 236us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:29.982315 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:53:29.982323 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:29.982388 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:29.982462 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 14:53:29.982488 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:53:29.982718 23953 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:53:29.990065 23953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 105b052b368b4d9189e5b0aba6021382: Generated new TSK 0
I20260928 14:53:29.990280 23953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:53:29.993446 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 23873
2026-09-28T14:53:29Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1341 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:53:30 dist-test-slave-pxl6 krb5kdc[23975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:53:30 dist-test-slave-pxl6 krb5kdc[23975](info): set up 2 sockets
Sep 28 14:53:30 dist-test-slave-pxl6 krb5kdc[23975](info): commencing operation
krb5kdc: starting...
W20260928 14:53:31.043535 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:53:31 dist-test-slave-pxl6 krb5kdc[23975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:53:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:53:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:53:31.167827 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:37575
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:35749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:37575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:31.252559 23988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:31.252815 23988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:31.252854 23988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:31.254057 23988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:53:31.254127 23988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:31.254153 23988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:31.254184 23988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:53:31.254207 23988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:53:31.255825 23988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:37575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:37575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:31.256116 23988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:31.256362 23988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:31.261634 23993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:31.261634 23994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:31.261634 23996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:31.261821 23988 server_base.cc:1034] running on GCE node
I20260928 14:53:31.262140 23988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:31.262486 23988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:31.263753 23988 hybrid_clock.cc:648] HybridClock initialized: now 1790607211263695 us; error 67 us; skew 500 ppm
Sep 28 14:53:31 dist-test-slave-pxl6 krb5kdc[23975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607211, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:31.266660 23988 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:53:31.267199 23988 webserver.cc:460] Webserver started at http://127.20.116.190:40255/ using document root <none> and password file <none>
I20260928 14:53:31.267640 23988 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:31.267799 23988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:31.267975 23988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:31.269125 23988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "6763de8ba67a4865b4454a524d510eb1"
format_stamp: "Formatted at 2026-09-28 14:53:31 on dist-test-slave-pxl6"
server_key: "d7f8faa7e93751ccbeb102ba692013a0"
server_key_iv: "e943e210596d8e18dd1dca15c754dc72"
server_key_version: "encryptionkey@0"
I20260928 14:53:31.269493 23988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "6763de8ba67a4865b4454a524d510eb1"
format_stamp: "Formatted at 2026-09-28 14:53:31 on dist-test-slave-pxl6"
server_key: "d7f8faa7e93751ccbeb102ba692013a0"
server_key_iv: "e943e210596d8e18dd1dca15c754dc72"
server_key_version: "encryptionkey@0"
I20260928 14:53:31.270951 23988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260928 14:53:31.271765 24003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:31.271951 23988 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:31.272060 23988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "6763de8ba67a4865b4454a524d510eb1"
format_stamp: "Formatted at 2026-09-28 14:53:31 on dist-test-slave-pxl6"
server_key: "d7f8faa7e93751ccbeb102ba692013a0"
server_key_iv: "e943e210596d8e18dd1dca15c754dc72"
server_key_version: "encryptionkey@0"
I20260928 14:53:31.272140 23988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:31.288609 23988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:31.290804 23988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:31.290980 23988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:31.295267 23988 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:37575
I20260928 14:53:31.295325 24055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:37575 every 8 connection(s)
I20260928 14:53:31.295734 23988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:53:31.296794 24056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:31.299312 24056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1: Bootstrap starting.
I20260928 14:53:31.299866 24056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:31.300115 24056 log.cc:824] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:31.300882 24056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1: No bootstrap required, opened a new log
I20260928 14:53:31.302515 24056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6763de8ba67a4865b4454a524d510eb1" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 37575 } }
I20260928 14:53:31.302697 24056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:31.302721 24056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6763de8ba67a4865b4454a524d510eb1, State: Initialized, Role: FOLLOWER
I20260928 14:53:31.302832 24056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6763de8ba67a4865b4454a524d510eb1" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 37575 } }
I20260928 14:53:31.302933 24056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:53:31.302969 24056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:53:31.303052 24056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:31.303359 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 23988
I20260928 14:53:31.303442 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:53:31.303645 20946 external_mini_cluster.cc:1428] Setting key fdd2d08dc31d7be6949b2890430a398a
I20260928 14:53:31.304149 24056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6763de8ba67a4865b4454a524d510eb1" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 37575 } }
I20260928 14:53:31.304299 24056 leader_election.cc:302] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6763de8ba67a4865b4454a524d510eb1; no voters:
I20260928 14:53:31.304492 24056 leader_election.cc:288] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:53:31.304567 24061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:31.304770 24061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1 [term 1 LEADER]: Becoming Leader. State: Replica: 6763de8ba67a4865b4454a524d510eb1, State: Running, Role: LEADER
I20260928 14:53:31.304785 24056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:31.304955 24061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6763de8ba67a4865b4454a524d510eb1" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 37575 } }
I20260928 14:53:31.305395 24062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6763de8ba67a4865b4454a524d510eb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6763de8ba67a4865b4454a524d510eb1" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 37575 } } }
I20260928 14:53:31.305480 24062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:31.306015 24063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6763de8ba67a4865b4454a524d510eb1. Latest consensus state: current_term: 1 leader_uuid: "6763de8ba67a4865b4454a524d510eb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6763de8ba67a4865b4454a524d510eb1" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 37575 } } }
I20260928 14:53:31.306098 24063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:31.306571 24070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:53:31.307178 24070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:53:31.308802 24070 catalog_manager.cc:1347] Generated new cluster ID: 4c7d6d37d0924b1e9581fe4e622e2e10
I20260928 14:53:31.308853 24070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:53:31 dist-test-slave-pxl6 krb5kdc[23975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:31.316447 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:31.304176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58404 (local address 127.20.116.190:37575)
0928 14:53:31.304379 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:31.304390 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:31.304427 (+ 37us) server_negotiation.cc:407] Connection header received
0928 14:53:31.304586 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:31.304594 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:31.304664 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:31.305901 (+ 1237us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:31.306072 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:31.307531 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:31.308666 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:31.309272 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:31.309393 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:31.312242 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:31.312247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:31.312247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:31.312260 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:31.314396 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:31.314839 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:31.314843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:31.314843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:31.315039 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:31.315310 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:31.315311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:31.315311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:31.315634 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:31.315782 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:31.315961 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 14:53:31.316230 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":91,"threads_started":1}
I20260928 14:53:31.318018 24070 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:53:31.318687 24070 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:53:31.324301 24070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6763de8ba67a4865b4454a524d510eb1: Generated new TSK 0
I20260928 14:53:31.324528 24070 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
I20260928 14:53:31.351907 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:37575
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:35749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:31.438448 24082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:31.438732 24082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:31.438758 24082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:31.439960 24082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:31.439999 24082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:31.440026 24082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:53:31.441524 24082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:37575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:31.441794 24082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:31.442054 24082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:31.447350 24090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:31.447367 24087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:31.447371 24088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:31.447697 24082 server_base.cc:1034] running on GCE node
I20260928 14:53:31.447893 24082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:31.448107 24082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:31.449250 24082 hybrid_clock.cc:648] HybridClock initialized: now 1790607211449238 us; error 29 us; skew 500 ppm
Sep 28 14:53:31 dist-test-slave-pxl6 krb5kdc[23975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607211, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:31.451990 24082 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:53:31.452450 24082 webserver.cc:460] Webserver started at http://127.20.116.129:34147/ using document root <none> and password file <none>
I20260928 14:53:31.452728 24082 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:31.452798 24082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:31.452965 24082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:31.454031 24082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "a067dfcb8d5449709252bc30ae0aecda"
format_stamp: "Formatted at 2026-09-28 14:53:31 on dist-test-slave-pxl6"
server_key: "ce89919f86cb72895930febff2f9fa01"
server_key_iv: "ebb5a57216300586c0a689e2e9b78da6"
server_key_version: "encryptionkey@0"
I20260928 14:53:31.454459 24082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "a067dfcb8d5449709252bc30ae0aecda"
format_stamp: "Formatted at 2026-09-28 14:53:31 on dist-test-slave-pxl6"
server_key: "ce89919f86cb72895930febff2f9fa01"
server_key_iv: "ebb5a57216300586c0a689e2e9b78da6"
server_key_version: "encryptionkey@0"
I20260928 14:53:31.455993 24082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:53:31.456933 24097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:31.457123 24082 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:31.457218 24082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "a067dfcb8d5449709252bc30ae0aecda"
format_stamp: "Formatted at 2026-09-28 14:53:31 on dist-test-slave-pxl6"
server_key: "ce89919f86cb72895930febff2f9fa01"
server_key_iv: "ebb5a57216300586c0a689e2e9b78da6"
server_key_version: "encryptionkey@0"
I20260928 14:53:31.457280 24082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:31.472263 24082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:31.474907 24082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:31.475034 24082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:31.475325 24082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:31.475847 24082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:31.475901 24082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:31.475929 24082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:31.475942 24082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:31.482524 24082 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:41011
I20260928 14:53:31.482645 24211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:41011 every 8 connection(s)
I20260928 14:53:31.483026 24082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:53:31.487391 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 24082
I20260928 14:53:31.487510 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:53:31.487816 20946 external_mini_cluster.cc:1428] Setting key e4a3bbb5ace158a3731ad495d8d3d02b
Sep 28 14:53:31 dist-test-slave-pxl6 krb5kdc[23975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607211, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:31.493860 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:31.483844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:35981 (local address 127.20.116.190:37575)
0928 14:53:31.483916 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:31.483919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:31.484197 (+ 278us) server_negotiation.cc:407] Connection header received
0928 14:53:31.484772 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:31.484773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:31.484790 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:31.484843 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:31.485855 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:31.486382 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:31.487715 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:31.487965 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:31.488044 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:31.490583 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:31.490586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:31.490586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:31.490595 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:31.492268 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:31.492680 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:31.492681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:31.492682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:31.492752 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:31.493000 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:31.493001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:31.493001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:31.493181 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:31.493270 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:31.493607 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 14:53:31.493711 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:53:31.493884 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:31.483827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:37575 (local address 127.20.116.129:35981)
0928 14:53:31.484028 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:31.484093 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:31.484640 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:31.484917 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:31.484921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:31.485527 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:31.485743 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:31.485749 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:31.486485 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:31.486487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:31.487609 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:31.487612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:31.488069 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:31.488071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:31.488269 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:31.488852 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:31.488861 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:31.490466 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:31.492357 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:31.492364 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:31.492366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:31.492599 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:31.492826 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:31.492826 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:31.492827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:31.492930 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:31.493275 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:31.493282 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:31.493430 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 14:53:31.493555 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 14:53:31.493694 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
I20260928 14:53:31.494572 24212 heartbeater.cc:347] Connected to a master server at 127.20.116.190:37575
I20260928 14:53:31.494700 24212 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:31.494900 24212 heartbeater.cc:511] Master 127.20.116.190:37575 requested a full tablet report, sending...
I20260928 14:53:31.495530 24020 ts_manager.cc:194] Registered new tserver with Master: a067dfcb8d5449709252bc30ae0aecda (127.20.116.129:41011)
I20260928 14:53:31.496206 24020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:35981
WARNING: no policy specified for kudu/127.20.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
I20260928 14:53:31.517001 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:37575
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:35749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:31.599469 24217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:31.599695 24217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:31.599720 24217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:31.600912 24217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:31.600957 24217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:31.600984 24217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:53:31.602607 24217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:37575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:31.602882 24217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:31.603132 24217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:31.608670 24222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:31.608651 24223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:31.608776 24217 server_base.cc:1034] running on GCE node
W20260928 14:53:31.608651 24225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:31.609159 24217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:31.609464 24217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:31.610599 24217 hybrid_clock.cc:648] HybridClock initialized: now 1790607211610599 us; error 35 us; skew 500 ppm
Sep 28 14:53:31 dist-test-slave-pxl6 krb5kdc[23975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607211, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:31.613339 24217 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:53:31.613891 24217 webserver.cc:460] Webserver started at http://127.20.116.130:45867/ using document root <none> and password file <none>
I20260928 14:53:31.614133 24217 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:31.614179 24217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:31.614331 24217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:31.615483 24217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "8573e475532a4e44bbd70518c31e9f00"
format_stamp: "Formatted at 2026-09-28 14:53:31 on dist-test-slave-pxl6"
server_key: "22deeefc59048bcf2c86ca5523f4cabd"
server_key_iv: "e714edad63002046983eb5d5095fafba"
server_key_version: "encryptionkey@0"
I20260928 14:53:31.615860 24217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "8573e475532a4e44bbd70518c31e9f00"
format_stamp: "Formatted at 2026-09-28 14:53:31 on dist-test-slave-pxl6"
server_key: "22deeefc59048bcf2c86ca5523f4cabd"
server_key_iv: "e714edad63002046983eb5d5095fafba"
server_key_version: "encryptionkey@0"
I20260928 14:53:31.617254 24217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:53:31.618111 24232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:31.618309 24217 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:31.618422 24217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "8573e475532a4e44bbd70518c31e9f00"
format_stamp: "Formatted at 2026-09-28 14:53:31 on dist-test-slave-pxl6"
server_key: "22deeefc59048bcf2c86ca5523f4cabd"
server_key_iv: "e714edad63002046983eb5d5095fafba"
server_key_version: "encryptionkey@0"
I20260928 14:53:31.618500 24217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:31.634630 24217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:31.637037 24217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:31.637203 24217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:31.637449 24217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:31.637773 24217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:31.637830 24217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:31.637876 24217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:31.637898 24217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:31.644229 24217 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:45145
I20260928 14:53:31.644285 24346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:45145 every 8 connection(s)
I20260928 14:53:31.644608 24217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:53:31 dist-test-slave-pxl6 krb5kdc[23975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607211, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:31.652271 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 24217
I20260928 14:53:31.652417 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:53:31.652667 20946 external_mini_cluster.cc:1428] Setting key 08f4c4d6732ea1e506ace07f09dee097
I20260928 14:53:31.655452 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:31.645391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:45469 (local address 127.20.116.190:37575)
0928 14:53:31.645466 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:31.645468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:31.645744 (+ 276us) server_negotiation.cc:407] Connection header received
0928 14:53:31.646289 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:31.646291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:31.646303 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:31.646370 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:31.647474 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:31.647974 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:31.649016 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:31.649217 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:31.649273 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:31.651661 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:31.651663 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:31.651663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:31.651672 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:31.653347 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:31.654193 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:31.654194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:31.654195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:31.654259 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:31.654638 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:31.654639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:31.654639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:31.654764 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:31.654836 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:31.655195 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 14:53:31.655321 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:53:31.655606 24349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:31.645360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:37575 (local address 127.20.116.130:45469)
0928 14:53:31.645552 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:31.645611 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:31.646148 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:31.646457 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:31.646460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:31.647123 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:31.647358 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:31.647364 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:31.648071 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:31.648072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:31.648930 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:31.648932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:31.649294 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:31.649294 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:31.649400 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:31.649952 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:31.649961 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:31.651542 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:31.653612 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:31.653622 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:31.653625 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:31.654052 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:31.654346 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:31.654347 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:31.654347 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:31.654509 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:31.654838 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:31.654845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:31.655032 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 14:53:31.655144 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 14:53:31.655343 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":112,"threads_started":1}
I20260928 14:53:31.656272 24347 heartbeater.cc:347] Connected to a master server at 127.20.116.190:37575
I20260928 14:53:31.656400 24347 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:31.656652 24347 heartbeater.cc:511] Master 127.20.116.190:37575 requested a full tablet report, sending...
I20260928 14:53:31.657109 24020 ts_manager.cc:194] Registered new tserver with Master: 8573e475532a4e44bbd70518c31e9f00 (127.20.116.130:45145)
I20260928 14:53:31.657676 24020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:45469
WARNING: no policy specified for kudu/127.20.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
I20260928 14:53:31.681599 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:37575
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:35749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:31.766784 24352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:31.767005 24352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:31.767030 24352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:31.768252 24352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:31.768304 24352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:31.768333 24352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:53:31.769838 24352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:37575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:31.770117 24352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:31.770370 24352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:31.775576 24358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:31.775606 24357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:31.775619 24360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:31.775910 24352 server_base.cc:1034] running on GCE node
I20260928 14:53:31.776144 24352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:31.776396 24352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:31.777560 24352 hybrid_clock.cc:648] HybridClock initialized: now 1790607211777545 us; error 36 us; skew 500 ppm
Sep 28 14:53:31 dist-test-slave-pxl6 krb5kdc[23975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607211, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:31.780189 24352 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:53:31.780682 24352 webserver.cc:460] Webserver started at http://127.20.116.131:35543/ using document root <none> and password file <none>
I20260928 14:53:31.780885 24352 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:31.780927 24352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:31.781023 24352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:31.782078 24352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "a15b231c46e14c97aba5485c1bb093ac"
format_stamp: "Formatted at 2026-09-28 14:53:31 on dist-test-slave-pxl6"
server_key: "f84907d68cc1651a0acf8490261ca17d"
server_key_iv: "a9ea9f3f528bcc0b309017399365ac86"
server_key_version: "encryptionkey@0"
I20260928 14:53:31.782389 24352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "a15b231c46e14c97aba5485c1bb093ac"
format_stamp: "Formatted at 2026-09-28 14:53:31 on dist-test-slave-pxl6"
server_key: "f84907d68cc1651a0acf8490261ca17d"
server_key_iv: "a9ea9f3f528bcc0b309017399365ac86"
server_key_version: "encryptionkey@0"
I20260928 14:53:31.783823 24352 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:53:31.784672 24367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:31.784876 24352 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:31.784955 24352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "a15b231c46e14c97aba5485c1bb093ac"
format_stamp: "Formatted at 2026-09-28 14:53:31 on dist-test-slave-pxl6"
server_key: "f84907d68cc1651a0acf8490261ca17d"
server_key_iv: "a9ea9f3f528bcc0b309017399365ac86"
server_key_version: "encryptionkey@0"
I20260928 14:53:31.785015 24352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:31.793970 24352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:31.796528 24352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:31.796681 24352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:31.796943 24352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:31.797273 24352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:31.797312 24352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:31.797335 24352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:31.797384 24352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:31.803519 24352 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:38091
I20260928 14:53:31.803588 24480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:38091 every 8 connection(s)
I20260928 14:53:31.803915 24352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:53:31.806954 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 24352
I20260928 14:53:31.807085 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:53:31.807276 20946 external_mini_cluster.cc:1428] Setting key d2632dfca6eb4f3020e5aeba0c368b57
Sep 28 14:53:31 dist-test-slave-pxl6 krb5kdc[23975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607211, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:31.814704 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:31.804711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:45427 (local address 127.20.116.190:37575)
0928 14:53:31.804775 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:31.804777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:31.805026 (+ 249us) server_negotiation.cc:407] Connection header received
0928 14:53:31.805606 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:31.805607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:31.805621 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:31.805681 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:31.807041 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:31.807766 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:31.809209 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:31.809403 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:31.809464 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:31.811875 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:31.811877 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:31.811877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:31.811889 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:31.813181 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:31.813608 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:31.813609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:31.813609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:31.813675 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:31.813946 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:31.813947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:31.813947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:31.814075 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:31.814182 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:31.814484 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 14:53:31.814594 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:53:31.814965 24483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:31.804719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:37575 (local address 127.20.116.131:45427)
0928 14:53:31.804896 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:31.804935 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:31.805492 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:31.805762 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:31.805766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:31.806492 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:31.806931 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:31.806937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:31.808011 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:31.808012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:31.809120 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:31.809123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:31.809489 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:31.809490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:31.809643 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:31.810151 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:31.810159 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:31.811757 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:31.813273 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:31.813282 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:31.813284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:31.813525 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:31.813747 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:31.813748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:31.813749 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:31.813868 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:31.814180 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:31.814188 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:31.814343 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 14:53:31.814460 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 14:53:31.814684 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260928 14:53:31.815603 24481 heartbeater.cc:347] Connected to a master server at 127.20.116.190:37575
I20260928 14:53:31.815712 24481 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:31.815871 24481 heartbeater.cc:511] Master 127.20.116.190:37575 requested a full tablet report, sending...
I20260928 14:53:31.816293 24020 ts_manager.cc:194] Registered new tserver with Master: a15b231c46e14c97aba5485c1bb093ac (127.20.116.131:38091)
I20260928 14:53:31.816702 24020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:45427
I20260928 14:53:31.819393 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:53:31 dist-test-slave-pxl6 krb5kdc[23975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607211, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 14:53:31 dist-test-slave-pxl6 krb5kdc[23975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607211, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:31.844020 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:31.836155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58408 (local address 127.20.116.190:37575)
0928 14:53:31.836265 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:31.836267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:31.836384 (+ 117us) server_negotiation.cc:407] Connection header received
0928 14:53:31.836504 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:31.836505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:31.836518 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:31.836589 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:31.837225 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:31.837746 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:31.838552 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:31.838756 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:31.838846 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:31.841239 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:31.841240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:31.841240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:31.841248 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:31.842707 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:31.843057 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:31.843059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:31.843059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:31.843117 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:31.843364 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:31.843364 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:31.843365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:31.843528 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:31.843607 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:31.843809 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 14:53:31.843906 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 14:53:31.844293 24020 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58408
I20260928 14:53:31.844945 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 24082
I20260928 14:53:31.850229 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 24217
I20260928 14:53:31.855245 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 24352
I20260928 14:53:31.859665 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 23988
2026-09-28T14:53:31Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1867 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:53:31 dist-test-slave-pxl6 krb5kdc[24493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:53:31 dist-test-slave-pxl6 krb5kdc[24493](info): set up 2 sockets
Sep 28 14:53:31 dist-test-slave-pxl6 krb5kdc[24493](info): commencing operation
krb5kdc: starting...
W20260928 14:53:32.927613 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:53:33 dist-test-slave-pxl6 krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:53:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:53:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:53:33.054203 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:45405
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:45667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:45405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:33.139762 24506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:33.139985 24506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:33.140008 24506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:33.141130 24506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:53:33.141165 24506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:33.141179 24506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:33.141191 24506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:53:33.141203 24506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:53:33.142705 24506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:45667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:45405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:45405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:33.142910 24506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:33.143096 24506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:33.148162 24514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:33.148308 24506 server_base.cc:1034] running on GCE node
W20260928 14:53:33.148201 24512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:33.148200 24511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:33.148708 24506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:33.149071 24506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:33.150270 24506 hybrid_clock.cc:648] HybridClock initialized: now 1790607213150251 us; error 41 us; skew 500 ppm
Sep 28 14:53:33 dist-test-slave-pxl6 krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607213, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:33.152886 24506 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:53:33.153314 24506 webserver.cc:460] Webserver started at http://127.20.116.190:39163/ using document root <none> and password file <none>
I20260928 14:53:33.153498 24506 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:33.153538 24506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:33.153627 24506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:33.154760 24506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "3b32738f14b849b3935fb051cb9c4c32"
format_stamp: "Formatted at 2026-09-28 14:53:33 on dist-test-slave-pxl6"
server_key: "4eacfb46c97dd15accd1b2a3c560ffb8"
server_key_iv: "aea0cd27f9925d988fb816b7496f18f1"
server_key_version: "encryptionkey@0"
I20260928 14:53:33.155076 24506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "3b32738f14b849b3935fb051cb9c4c32"
format_stamp: "Formatted at 2026-09-28 14:53:33 on dist-test-slave-pxl6"
server_key: "4eacfb46c97dd15accd1b2a3c560ffb8"
server_key_iv: "aea0cd27f9925d988fb816b7496f18f1"
server_key_version: "encryptionkey@0"
I20260928 14:53:33.156425 24506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:53:33.157204 24521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:33.157401 24506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:33.157497 24506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "3b32738f14b849b3935fb051cb9c4c32"
format_stamp: "Formatted at 2026-09-28 14:53:33 on dist-test-slave-pxl6"
server_key: "4eacfb46c97dd15accd1b2a3c560ffb8"
server_key_iv: "aea0cd27f9925d988fb816b7496f18f1"
server_key_version: "encryptionkey@0"
I20260928 14:53:33.157552 24506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:33.193682 24506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:33.196533 24506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:33.196662 24506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:33.200961 24506 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:45405
I20260928 14:53:33.200996 24574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:45405 every 8 connection(s)
I20260928 14:53:33.201570 24506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:53:33.202412 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:33.204943 24575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32: Bootstrap starting.
I20260928 14:53:33.205639 24575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:33.205950 24575 log.cc:824] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:33.206859 24575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32: No bootstrap required, opened a new log
I20260928 14:53:33.208657 24575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b32738f14b849b3935fb051cb9c4c32" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 45405 } }
I20260928 14:53:33.208837 24575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:33.208887 24575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b32738f14b849b3935fb051cb9c4c32, State: Initialized, Role: FOLLOWER
I20260928 14:53:33.209049 24575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b32738f14b849b3935fb051cb9c4c32" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 45405 } }
I20260928 14:53:33.209137 24575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:53:33.209184 24575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:53:33.209239 24575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:33.210091 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 24506
I20260928 14:53:33.210216 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:53:33.210414 20946 external_mini_cluster.cc:1428] Setting key 6486d16ce357fb70e6fb9889ef4ad592
I20260928 14:53:33.210433 24575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b32738f14b849b3935fb051cb9c4c32" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 45405 } }
I20260928 14:53:33.210574 24575 leader_election.cc:302] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b32738f14b849b3935fb051cb9c4c32; no voters:
I20260928 14:53:33.210788 24575 leader_election.cc:288] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:53:33.210826 24578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:33.210970 24578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32 [term 1 LEADER]: Becoming Leader. State: Replica: 3b32738f14b849b3935fb051cb9c4c32, State: Running, Role: LEADER
I20260928 14:53:33.211071 24578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b32738f14b849b3935fb051cb9c4c32" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 45405 } }
I20260928 14:53:33.211283 24575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:33.211403 24578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b32738f14b849b3935fb051cb9c4c32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b32738f14b849b3935fb051cb9c4c32" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 45405 } } }
I20260928 14:53:33.211580 24578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32 [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:33.211767 24578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b32738f14b849b3935fb051cb9c4c32. Latest consensus state: current_term: 1 leader_uuid: "3b32738f14b849b3935fb051cb9c4c32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b32738f14b849b3935fb051cb9c4c32" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 45405 } } }
I20260928 14:53:33.211894 24578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32 [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:33.212224 24584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:53:33.212816 24584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:53:33.214716 24584 catalog_manager.cc:1347] Generated new cluster ID: 6500c4a3ac5b4e15b6fe3a7cea010b34
I20260928 14:53:33.214778 24584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:53:33 dist-test-slave-pxl6 krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:33.221182 24584 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:53:33.222050 24584 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:53:33.222260 24582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:33.211033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56778 (local address 127.20.116.190:45405)
0928 14:53:33.211367 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:33.211379 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:33.211422 (+ 43us) server_negotiation.cc:407] Connection header received
0928 14:53:33.211620 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:33.211628 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:33.211690 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:33.211843 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:33.212698 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:33.214150 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:33.215126 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:33.215585 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:33.215709 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:33.218090 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:33.218095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:33.218095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:33.218116 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:33.220413 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:33.220859 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:33.220861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:33.220862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:33.220958 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:33.221251 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:33.221252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:33.221252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:33.221493 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:33.221758 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:33.221836 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 14:53:33.222026 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":274,"thread_start_us":67,"threads_started":1}
I20260928 14:53:33.230005 24584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b32738f14b849b3935fb051cb9c4c32: Generated new TSK 0
I20260928 14:53:33.230273 24584 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
I20260928 14:53:33.266338 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:45405
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:45667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:33.367817 24601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:33.368038 24601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:33.368065 24601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:33.369287 24601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:33.369329 24601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:33.369359 24601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:53:33.370995 24601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:45667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:45405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:33.371263 24601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:33.371516 24601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:33.377537 24607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:33.377614 24606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:33.377803 24601 server_base.cc:1034] running on GCE node
W20260928 14:53:33.377537 24609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:33.378101 24601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:33.378322 24601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:33.379474 24601 hybrid_clock.cc:648] HybridClock initialized: now 1790607213379447 us; error 45 us; skew 500 ppm
Sep 28 14:53:33 dist-test-slave-pxl6 krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607213, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:33.386724 24601 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:53:33.387182 24601 webserver.cc:460] Webserver started at http://127.20.116.129:40051/ using document root <none> and password file <none>
I20260928 14:53:33.387398 24601 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:33.387440 24601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:33.387537 24601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:33.388617 24601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "a68f358eef6c45d1bb39e4cc91495fd8"
format_stamp: "Formatted at 2026-09-28 14:53:33 on dist-test-slave-pxl6"
server_key: "a0ed745b6a14c8b8784c193a34ef2f02"
server_key_iv: "696ffe455f6a9afa039cbf6d5a9c4d39"
server_key_version: "encryptionkey@0"
I20260928 14:53:33.388932 24601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "a68f358eef6c45d1bb39e4cc91495fd8"
format_stamp: "Formatted at 2026-09-28 14:53:33 on dist-test-slave-pxl6"
server_key: "a0ed745b6a14c8b8784c193a34ef2f02"
server_key_iv: "696ffe455f6a9afa039cbf6d5a9c4d39"
server_key_version: "encryptionkey@0"
I20260928 14:53:33.390376 24601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:53:33.391301 24616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:33.391479 24601 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:33.391569 24601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "a68f358eef6c45d1bb39e4cc91495fd8"
format_stamp: "Formatted at 2026-09-28 14:53:33 on dist-test-slave-pxl6"
server_key: "a0ed745b6a14c8b8784c193a34ef2f02"
server_key_iv: "696ffe455f6a9afa039cbf6d5a9c4d39"
server_key_version: "encryptionkey@0"
I20260928 14:53:33.391630 24601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:33.429071 24601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:33.431315 24601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:33.431428 24601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:33.431628 24601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:33.431974 24601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:33.432010 24601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:33.432034 24601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:33.432085 24601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:33.438755 24601 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:45401
I20260928 14:53:33.438822 24731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:45401 every 8 connection(s)
I20260928 14:53:33.439235 24601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:53:33.443084 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 24601
I20260928 14:53:33.443203 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:53:33.443370 20946 external_mini_cluster.cc:1428] Setting key 8ac75e71403ee292526633101ec50528
Sep 28 14:53:33 dist-test-slave-pxl6 krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607213, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:33.451097 24582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:33.440142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:58761 (local address 127.20.116.190:45405)
0928 14:53:33.440221 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:33.440223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:33.440421 (+ 198us) server_negotiation.cc:407] Connection header received
0928 14:53:33.441466 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:33.441467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:33.441481 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:33.441554 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:33.442277 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:33.442867 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:33.444097 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:33.444313 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:33.444374 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:33.447307 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:33.447309 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:33.447309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:33.447317 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:33.449272 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:33.449800 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:33.449801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:33.449802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:33.449872 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:33.450194 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:33.450195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:33.450195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:33.450379 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:33.450461 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:33.450842 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 14:53:33.450948 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:53:33.451133 24734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:33.440112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:45405 (local address 127.20.116.129:58761)
0928 14:53:33.440287 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:33.440327 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:33.441345 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:33.441638 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:33.441641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:33.441909 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:33.442144 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:33.442148 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:33.443000 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:33.443001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:33.443987 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:33.443990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:33.444411 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:33.444412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:33.444584 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:33.445185 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:33.445196 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:33.447176 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:33.449384 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:33.449393 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:33.449395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:33.449700 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:33.449972 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:33.449973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:33.449973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:33.450111 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:33.450459 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:33.450466 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:33.450684 (+ 218us) client_negotiation.cc:769] Sending connection context
0928 14:53:33.450781 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 14:53:33.450938 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260928 14:53:33.451804 24732 heartbeater.cc:347] Connected to a master server at 127.20.116.190:45405
I20260928 14:53:33.451907 24732 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:33.452129 24732 heartbeater.cc:511] Master 127.20.116.190:45405 requested a full tablet report, sending...
I20260928 14:53:33.452757 24538 ts_manager.cc:194] Registered new tserver with Master: a68f358eef6c45d1bb39e4cc91495fd8 (127.20.116.129:45401)
WARNING: no policy specified for kudu/127.20.116.130@KRBTEST.COM; defaulting to no policy
I20260928 14:53:33.453393 24538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:58761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
I20260928 14:53:33.473912 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:45405
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:45667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:33.557542 24737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:33.557811 24737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:33.557852 24737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:33.559072 24737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:33.559150 24737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:33.559255 24737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:53:33.560912 24737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:45667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:45405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:33.561213 24737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:33.561460 24737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:33.566941 24745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:33.566941 24742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:33.566941 24743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:33.566959 24737 server_base.cc:1034] running on GCE node
I20260928 14:53:33.567382 24737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:33.567595 24737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:33.568733 24737 hybrid_clock.cc:648] HybridClock initialized: now 1790607213568717 us; error 35 us; skew 500 ppm
Sep 28 14:53:33 dist-test-slave-pxl6 krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607213, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:33.571564 24737 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:53:33.571954 24737 webserver.cc:460] Webserver started at http://127.20.116.130:39381/ using document root <none> and password file <none>
I20260928 14:53:33.572144 24737 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:33.572185 24737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:33.572271 24737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:33.573285 24737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "533c9f7997e543b3a955405bca2b1bc0"
format_stamp: "Formatted at 2026-09-28 14:53:33 on dist-test-slave-pxl6"
server_key: "9843a67fcdd496423366d4bd6df58659"
server_key_iv: "be8ef8e860d36eec8fa47e2a7636fb5a"
server_key_version: "encryptionkey@0"
I20260928 14:53:33.573583 24737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "533c9f7997e543b3a955405bca2b1bc0"
format_stamp: "Formatted at 2026-09-28 14:53:33 on dist-test-slave-pxl6"
server_key: "9843a67fcdd496423366d4bd6df58659"
server_key_iv: "be8ef8e860d36eec8fa47e2a7636fb5a"
server_key_version: "encryptionkey@0"
I20260928 14:53:33.574884 24737 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:53:33.575570 24752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:33.575764 24737 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:53:33.575856 24737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "533c9f7997e543b3a955405bca2b1bc0"
format_stamp: "Formatted at 2026-09-28 14:53:33 on dist-test-slave-pxl6"
server_key: "9843a67fcdd496423366d4bd6df58659"
server_key_iv: "be8ef8e860d36eec8fa47e2a7636fb5a"
server_key_version: "encryptionkey@0"
I20260928 14:53:33.575928 24737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:33.600960 24737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:33.603569 24737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:33.603718 24737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:33.604014 24737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:33.604363 24737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:33.604398 24737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:33.604422 24737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:33.604472 24737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:33.610810 24737 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:45829
I20260928 14:53:33.610888 24866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:45829 every 8 connection(s)
I20260928 14:53:33.611272 24737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:53:33 dist-test-slave-pxl6 krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607213, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:33.619649 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 24737
I20260928 14:53:33.619771 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:53:33.619961 20946 external_mini_cluster.cc:1428] Setting key b2698c55e7febc68194cfe9747dfac73
I20260928 14:53:33.621934 24582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:33.612174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:40939 (local address 127.20.116.190:45405)
0928 14:53:33.612221 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:33.612223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:33.612521 (+ 298us) server_negotiation.cc:407] Connection header received
0928 14:53:33.613569 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:33.613570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:33.613582 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:33.613634 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:33.614480 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:33.615129 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:33.616129 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:33.616335 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:33.616398 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:33.618692 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:33.618693 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:33.618693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:33.618701 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:33.620422 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:33.620907 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:33.620908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:33.620908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:33.620976 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:33.621257 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:33.621258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:33.621258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:33.621384 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:33.621466 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:33.621755 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 14:53:33.621844 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:53:33.622167 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:33.612179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:45405 (local address 127.20.116.130:40939)
0928 14:53:33.612365 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:33.612415 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:33.613455 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:33.613711 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:33.613714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:33.614007 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:33.614224 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:33.614229 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:33.615218 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:33.615219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:33.616039 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:33.616041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:33.616418 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:33.616419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:33.616533 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:33.617042 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:33.617051 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:33.618510 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:33.620497 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:33.620505 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:33.620507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:33.620808 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:33.621065 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:33.621066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:33.621067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:33.621179 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:33.621455 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:33.621462 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:33.621613 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 14:53:33.621722 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 14:53:33.621886 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260928 14:53:33.622752 24867 heartbeater.cc:347] Connected to a master server at 127.20.116.190:45405
I20260928 14:53:33.622875 24867 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:33.623070 24867 heartbeater.cc:511] Master 127.20.116.190:45405 requested a full tablet report, sending...
I20260928 14:53:33.623507 24538 ts_manager.cc:194] Registered new tserver with Master: 533c9f7997e543b3a955405bca2b1bc0 (127.20.116.130:45829)
I20260928 14:53:33.623915 24538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:40939
WARNING: no policy specified for kudu/127.20.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
I20260928 14:53:33.645596 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:45405
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:45667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:33.731824 24872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:33.732043 24872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:33.732067 24872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:33.733194 24872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:33.733228 24872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:33.733255 24872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:53:33.734825 24872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:45667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:45405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:33.735024 24872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:33.735210 24872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:33.740470 24878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:33.740499 24872 server_base.cc:1034] running on GCE node
W20260928 14:53:33.740466 24877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:33.740466 24880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:33.740935 24872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:33.741153 24872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:33.742297 24872 hybrid_clock.cc:648] HybridClock initialized: now 1790607213742276 us; error 39 us; skew 500 ppm
Sep 28 14:53:33 dist-test-slave-pxl6 krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607213, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:33.745107 24872 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:53:33.745579 24872 webserver.cc:460] Webserver started at http://127.20.116.131:44413/ using document root <none> and password file <none>
I20260928 14:53:33.745846 24872 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:33.745896 24872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:33.746028 24872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:33.747179 24872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "6d13c93acbb54276b7502ed4e6b3dfec"
format_stamp: "Formatted at 2026-09-28 14:53:33 on dist-test-slave-pxl6"
server_key: "3a47e37a9a75ae481a20c70f2ec13294"
server_key_iv: "2105375e5cc90e8997ddeeff58fac759"
server_key_version: "encryptionkey@0"
I20260928 14:53:33.747553 24872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "6d13c93acbb54276b7502ed4e6b3dfec"
format_stamp: "Formatted at 2026-09-28 14:53:33 on dist-test-slave-pxl6"
server_key: "3a47e37a9a75ae481a20c70f2ec13294"
server_key_iv: "2105375e5cc90e8997ddeeff58fac759"
server_key_version: "encryptionkey@0"
I20260928 14:53:33.748967 24872 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 14:53:33.749851 24887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:33.750099 24872 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:33.750201 24872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "6d13c93acbb54276b7502ed4e6b3dfec"
format_stamp: "Formatted at 2026-09-28 14:53:33 on dist-test-slave-pxl6"
server_key: "3a47e37a9a75ae481a20c70f2ec13294"
server_key_iv: "2105375e5cc90e8997ddeeff58fac759"
server_key_version: "encryptionkey@0"
I20260928 14:53:33.750290 24872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:33.773660 24872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:33.776114 24872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:33.776273 24872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:33.776551 24872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:33.776901 24872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:33.776935 24872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:33.776959 24872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:33.776989 24872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:33.783556 24872 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:33481
I20260928 14:53:33.783707 25000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:33481 every 8 connection(s)
I20260928 14:53:33.783998 24872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:53:33.790895 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 24872
I20260928 14:53:33.790982 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
Sep 28 14:53:33 dist-test-slave-pxl6 krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607213, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:33.791198 20946 external_mini_cluster.cc:1428] Setting key 106dc950b05f8462300aed2504eb18be
I20260928 14:53:33.795339 24582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:33.784872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:57567 (local address 127.20.116.190:45405)
0928 14:53:33.784943 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:33.784945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:33.785173 (+ 228us) server_negotiation.cc:407] Connection header received
0928 14:53:33.786161 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:33.786162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:33.786175 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:33.786238 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:33.786902 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:33.787410 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:33.788524 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:33.788766 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:33.788864 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:33.791842 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:33.791844 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:33.791844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:33.791851 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:33.793434 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:33.794204 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:33.794207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:33.794207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:33.794277 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:33.794525 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:33.794526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:33.794526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:33.794721 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:33.794779 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:33.795158 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 14:53:33.795228 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:53:33.795671 25003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:33.784831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:45405 (local address 127.20.116.131:57567)
0928 14:53:33.785030 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:33.785074 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:33.786054 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:33.786298 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:33.786302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:33.786598 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:33.786817 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:33.786822 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:33.787547 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:33.787548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:33.788435 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:33.788438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:33.788903 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:33.788905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:33.789104 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:33.789822 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:33.789835 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:33.791710 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:33.793581 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:33.793713 (+ 132us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:33.793716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:33.794072 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:33.794345 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:33.794345 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:33.794346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:33.794463 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:33.794787 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:33.794794 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:33.795009 (+ 215us) client_negotiation.cc:769] Sending connection context
0928 14:53:33.795128 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 14:53:33.795312 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":122,"threads_started":1}
I20260928 14:53:33.796348 25001 heartbeater.cc:347] Connected to a master server at 127.20.116.190:45405
I20260928 14:53:33.796463 25001 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:33.796701 25001 heartbeater.cc:511] Master 127.20.116.190:45405 requested a full tablet report, sending...
I20260928 14:53:33.797150 24538 ts_manager.cc:194] Registered new tserver with Master: 6d13c93acbb54276b7502ed4e6b3dfec (127.20.116.131:33481)
I20260928 14:53:33.797626 24538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:57567
I20260928 14:53:33.803309 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:53:33 dist-test-slave-pxl6 krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:53:33 dist-test-slave-pxl6 krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.129@KRBTEST.COM
I20260928 14:53:33.827019 25009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:33.819242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.20.116.129:45401)
0928 14:53:33.819430 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:33.819436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:33.819447 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:53:33.819476 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:33.819478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:33.819494 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:33.819563 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:33.820138 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:33.820885 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:33.821592 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:33.821947 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:33.822004 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:33.824248 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:33.824258 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:33.824258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:33.824265 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:33.825931 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:33.826229 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:33.826232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:33.826232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:33.826287 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:33.826433 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:33.826434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:33.826434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:33.826614 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:33.826745 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:33.826818 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 14:53:33.826880 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":93,"threads_started":1}
Sep 28 14:53:33 dist-test-slave-pxl6 krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:33.835335 24582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:33.828376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56792 (local address 127.20.116.190:45405)
0928 14:53:33.828455 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:33.828458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:33.828493 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:53:33.828576 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:33.828577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:33.828589 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:33.828640 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:33.829118 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:33.829648 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:33.830378 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:33.830678 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:33.830760 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:33.832840 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:33.832842 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:33.832842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:33.832850 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:33.834111 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:33.834472 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:33.834473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:33.834473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:33.834564 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:33.834757 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:33.834759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:33.834759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:33.834922 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:33.835035 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:33.835141 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 14:53:33.835225 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 14:53:33.835611 24538 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56792
I20260928 14:53:33.836009 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 24601
I20260928 14:53:33.841113 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 24737
I20260928 14:53:33.845914 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 24872
I20260928 14:53:33.850369 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 24506
2026-09-28T14:53:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1990 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:53:33 dist-test-slave-pxl6 krb5kdc[25014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:53:33 dist-test-slave-pxl6 krb5kdc[25014](info): set up 2 sockets
Sep 28 14:53:33 dist-test-slave-pxl6 krb5kdc[25014](info): commencing operation
krb5kdc: starting...
W20260928 14:53:34.895649 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:53:34 dist-test-slave-pxl6 krb5kdc[25014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:53:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:53:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:53:35.021087 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:45801
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:35409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:45801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:35.108165 25027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:35.108426 25027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:35.108464 25027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:35.109685 25027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:53:35.109754 25027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:35.109777 25027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:35.109798 25027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:53:35.109829 25027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:53:35.111406 25027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:45801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:45801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:35.111706 25027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:35.111944 25027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:35.117223 25035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:35.117208 25032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:35.117267 25033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:35.117395 25027 server_base.cc:1034] running on GCE node
I20260928 14:53:35.117812 25027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:35.118142 25027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:35.119333 25027 hybrid_clock.cc:648] HybridClock initialized: now 1790607215119324 us; error 38 us; skew 500 ppm
Sep 28 14:53:35 dist-test-slave-pxl6 krb5kdc[25014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607215, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:35.121807 25027 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:53:35.122207 25027 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:35.122287 25027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:35.122455 25027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:35.123610 25027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "47c7bb19ff9c40bca24d04cd6db2369c"
format_stamp: "Formatted at 2026-09-28 14:53:35 on dist-test-slave-pxl6"
server_key: "c7cce2e34d64fd568d57bed390b8813e"
server_key_iv: "b53fd59cd6b15db3e7fd3f59a12cc6ea"
server_key_version: "encryptionkey@0"
I20260928 14:53:35.123989 25027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "47c7bb19ff9c40bca24d04cd6db2369c"
format_stamp: "Formatted at 2026-09-28 14:53:35 on dist-test-slave-pxl6"
server_key: "c7cce2e34d64fd568d57bed390b8813e"
server_key_iv: "b53fd59cd6b15db3e7fd3f59a12cc6ea"
server_key_version: "encryptionkey@0"
I20260928 14:53:35.125346 25027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:53:35.126080 25041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:35.126273 25027 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:53:35.126386 25027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "47c7bb19ff9c40bca24d04cd6db2369c"
format_stamp: "Formatted at 2026-09-28 14:53:35 on dist-test-slave-pxl6"
server_key: "c7cce2e34d64fd568d57bed390b8813e"
server_key_iv: "b53fd59cd6b15db3e7fd3f59a12cc6ea"
server_key_version: "encryptionkey@0"
I20260928 14:53:35.126463 25027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:35.142275 25027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:35.144613 25027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:35.144908 25027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:35.149401 25027 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:45801
I20260928 14:53:35.149432 25093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:45801 every 8 connection(s)
I20260928 14:53:35.149791 25027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:53:35.150823 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:35.153292 25094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c: Bootstrap starting.
I20260928 14:53:35.153934 25094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:35.154201 25094 log.cc:824] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:35.155053 25094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c: No bootstrap required, opened a new log
I20260928 14:53:35.156250 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 25027
I20260928 14:53:35.156363 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:53:35.156560 20946 external_mini_cluster.cc:1428] Setting key ede6c8c9674ed77ca77d94f9ba92ab14
I20260928 14:53:35.157280 25094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47c7bb19ff9c40bca24d04cd6db2369c" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 45801 } }
I20260928 14:53:35.157470 25094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:35.157518 25094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47c7bb19ff9c40bca24d04cd6db2369c, State: Initialized, Role: FOLLOWER
I20260928 14:53:35.157656 25094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47c7bb19ff9c40bca24d04cd6db2369c" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 45801 } }
I20260928 14:53:35.157757 25094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:53:35.157802 25094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:53:35.157851 25094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:35.158715 25094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47c7bb19ff9c40bca24d04cd6db2369c" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 45801 } }
I20260928 14:53:35.158849 25094 leader_election.cc:302] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47c7bb19ff9c40bca24d04cd6db2369c; no voters:
I20260928 14:53:35.159037 25094 leader_election.cc:288] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:53:35.159126 25099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:35.159317 25099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c [term 1 LEADER]: Becoming Leader. State: Replica: 47c7bb19ff9c40bca24d04cd6db2369c, State: Running, Role: LEADER
I20260928 14:53:35.159328 25094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:35.159439 25099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47c7bb19ff9c40bca24d04cd6db2369c" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 45801 } }
I20260928 14:53:35.159860 25099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47c7bb19ff9c40bca24d04cd6db2369c. Latest consensus state: current_term: 1 leader_uuid: "47c7bb19ff9c40bca24d04cd6db2369c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47c7bb19ff9c40bca24d04cd6db2369c" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 45801 } } }
I20260928 14:53:35.159866 25100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47c7bb19ff9c40bca24d04cd6db2369c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47c7bb19ff9c40bca24d04cd6db2369c" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 45801 } } }
I20260928 14:53:35.160045 25099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:35.160053 25100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:35.160337 25108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:53:35.160928 25108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:53:35.162752 25108 catalog_manager.cc:1347] Generated new cluster ID: ab00012c8be24791855921fa58739033
I20260928 14:53:35.162807 25108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:53:35 dist-test-slave-pxl6 krb5kdc[25014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:35.169719 25098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:35.157181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40318 (local address 127.20.116.190:45801)
0928 14:53:35.157379 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:35.157387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:35.157415 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:53:35.157834 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:35.157844 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:35.157916 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:35.158082 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:35.158951 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.160537 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.161819 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.162463 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.162626 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:35.165558 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:35.165563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:35.165563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:35.165585 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:35.167841 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:35.168259 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:35.168263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:35.168264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:35.168412 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:35.168626 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:35.168627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:35.168627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:35.168988 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:35.169194 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:35.169288 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 14:53:35.169472 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
I20260928 14:53:35.176645 25108 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:53:35.177431 25108 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:53:35.191224 25108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c: Generated new TSK 0
I20260928 14:53:35.191484 25108 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
I20260928 14:53:35.226351 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:45801
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:35409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:35.312919 25120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:35.313158 25120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:35.313186 25120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:35.314479 25120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:35.314524 25120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:35.314625 25120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:53:35.316166 25120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:45801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:35.316378 25120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:35.316587 25120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:35.321591 25128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:35.321614 25126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:35.321732 25120 server_base.cc:1034] running on GCE node
W20260928 14:53:35.321688 25125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:35.322022 25120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:35.322305 25120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:35.323472 25120 hybrid_clock.cc:648] HybridClock initialized: now 1790607215323460 us; error 50 us; skew 500 ppm
Sep 28 14:53:35 dist-test-slave-pxl6 krb5kdc[25014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607215, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:35.326236 25120 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:53:35.326658 25120 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:35.326733 25120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:35.326889 25120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:35.327991 25120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "dc42141fe76f41aab49a033f121d13b0"
format_stamp: "Formatted at 2026-09-28 14:53:35 on dist-test-slave-pxl6"
server_key: "d566a9305cdf5a30192003e8f8c68abf"
server_key_iv: "32c739a3b0347b2fac8d28eb13de553c"
server_key_version: "encryptionkey@0"
I20260928 14:53:35.328374 25120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "dc42141fe76f41aab49a033f121d13b0"
format_stamp: "Formatted at 2026-09-28 14:53:35 on dist-test-slave-pxl6"
server_key: "d566a9305cdf5a30192003e8f8c68abf"
server_key_iv: "32c739a3b0347b2fac8d28eb13de553c"
server_key_version: "encryptionkey@0"
I20260928 14:53:35.329895 25120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:53:35.330958 25134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:35.331161 25120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:53:35.331269 25120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "dc42141fe76f41aab49a033f121d13b0"
format_stamp: "Formatted at 2026-09-28 14:53:35 on dist-test-slave-pxl6"
server_key: "d566a9305cdf5a30192003e8f8c68abf"
server_key_iv: "32c739a3b0347b2fac8d28eb13de553c"
server_key_version: "encryptionkey@0"
I20260928 14:53:35.331362 25120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:35.355195 25120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:35.357813 25120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:35.358012 25120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:35.358304 25120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:35.358697 25120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:35.358763 25120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:35.358810 25120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:35.358837 25120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:35.365139 25120 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:38107
I20260928 14:53:35.365181 25248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:38107 every 8 connection(s)
I20260928 14:53:35.365478 25120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:53:35 dist-test-slave-pxl6 krb5kdc[25014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607215, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:35.372382 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 25120
I20260928 14:53:35.372563 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:53:35.372792 20946 external_mini_cluster.cc:1428] Setting key ff4c831a76f5701a330a29c2d2eca095
I20260928 14:53:35.376171 25098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:35.366300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:44901 (local address 127.20.116.190:45801)
0928 14:53:35.366371 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:35.366374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:35.366663 (+ 289us) server_negotiation.cc:407] Connection header received
0928 14:53:35.367238 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:35.367240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:35.367255 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:35.367324 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:35.368394 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.368967 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.369997 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.370374 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.370446 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:35.372933 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:35.372934 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:35.372934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:35.372946 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:35.374510 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:35.375098 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:35.375100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:35.375100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:35.375178 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:35.375432 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:35.375433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:35.375434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:35.375589 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:35.375660 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:35.375931 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 14:53:35.376034 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:53:35.376209 25251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:35.366288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:45801 (local address 127.20.116.129:44901)
0928 14:53:35.366488 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:35.366573 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:35.367080 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:35.367419 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:35.367423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:35.367719 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:35.368269 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:35.368275 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.369067 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.369068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:35.369898 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:35.369901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.370477 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.370479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:35.370622 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:35.371234 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:35.371244 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:35.372836 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:35.374628 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:35.374638 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:35.374641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:35.375000 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:35.375250 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:35.375251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:35.375251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:35.375365 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:35.375654 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:35.375661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:35.375798 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 14:53:35.375886 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 14:53:35.376017 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":91,"threads_started":1}
I20260928 14:53:35.377092 25249 heartbeater.cc:347] Connected to a master server at 127.20.116.190:45801
I20260928 14:53:35.377213 25249 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:35.377408 25249 heartbeater.cc:511] Master 127.20.116.190:45801 requested a full tablet report, sending...
I20260928 14:53:35.377991 25058 ts_manager.cc:194] Registered new tserver with Master: dc42141fe76f41aab49a033f121d13b0 (127.20.116.129:38107)
I20260928 14:53:35.378683 25058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:44901
WARNING: no policy specified for kudu/127.20.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
I20260928 14:53:35.401985 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:45801
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:35409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:35.485404 25254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:35.485698 25254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:35.485746 25254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:35.487063 25254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:35.487138 25254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:35.487185 25254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:53:35.488859 25254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:45801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:35.489166 25254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:35.489430 25254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:35.494656 25262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:35.494635 25260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:35.494635 25259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:35.494760 25254 server_base.cc:1034] running on GCE node
I20260928 14:53:35.495190 25254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:35.495402 25254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:35.496626 25254 hybrid_clock.cc:648] HybridClock initialized: now 1790607215496565 us; error 35 us; skew 500 ppm
Sep 28 14:53:35 dist-test-slave-pxl6 krb5kdc[25014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607215, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:35.499397 25254 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:53:35.499879 25254 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:35.499934 25254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:35.500102 25254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:35.501195 25254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "acfd31e4126247afaa206c441de6bcf7"
format_stamp: "Formatted at 2026-09-28 14:53:35 on dist-test-slave-pxl6"
server_key: "874538754237b167b6abb42fc6d3cd26"
server_key_iv: "5ea704bc5b1bd446a21fc81c08bae800"
server_key_version: "encryptionkey@0"
I20260928 14:53:35.501556 25254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "acfd31e4126247afaa206c441de6bcf7"
format_stamp: "Formatted at 2026-09-28 14:53:35 on dist-test-slave-pxl6"
server_key: "874538754237b167b6abb42fc6d3cd26"
server_key_iv: "5ea704bc5b1bd446a21fc81c08bae800"
server_key_version: "encryptionkey@0"
I20260928 14:53:35.502961 25254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:53:35.503818 25268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:35.504004 25254 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:35.504113 25254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "acfd31e4126247afaa206c441de6bcf7"
format_stamp: "Formatted at 2026-09-28 14:53:35 on dist-test-slave-pxl6"
server_key: "874538754237b167b6abb42fc6d3cd26"
server_key_iv: "5ea704bc5b1bd446a21fc81c08bae800"
server_key_version: "encryptionkey@0"
I20260928 14:53:35.504190 25254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:35.524634 25254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:35.527252 25254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:35.527429 25254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:35.527683 25254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:35.528015 25254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:35.528070 25254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:35.528136 25254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:35.528164 25254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:35.534649 25254 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:39237
I20260928 14:53:35.534726 25382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:39237 every 8 connection(s)
I20260928 14:53:35.535045 25254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:53:35.537617 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 25254
I20260928 14:53:35.537760 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:53:35.537942 20946 external_mini_cluster.cc:1428] Setting key ad6f125f681d9b4d9c819e05ecf9e70c
Sep 28 14:53:35 dist-test-slave-pxl6 krb5kdc[25014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607215, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:35.546168 25098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:35.535946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:42769 (local address 127.20.116.190:45801)
0928 14:53:35.535999 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:35.536002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:35.536243 (+ 241us) server_negotiation.cc:407] Connection header received
0928 14:53:35.536845 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:35.536847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:35.536860 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:35.536922 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:35.538033 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.538739 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.539794 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.540122 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.540207 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:35.542898 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:35.542899 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:35.542900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:35.542910 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:35.544677 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:35.545104 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:35.545106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:35.545106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:35.545176 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:35.545423 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:35.545424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:35.545424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:35.545593 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:35.545672 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:35.545946 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 14:53:35.546051 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:53:35.546213 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:35.535916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:45801 (local address 127.20.116.130:42769)
0928 14:53:35.536106 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:35.536145 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:35.536719 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:35.537021 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:35.537025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:35.537306 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:35.537923 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:35.537930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.538834 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.538835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:35.539689 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:35.539692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.540225 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.540227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:35.540348 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:35.540968 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:35.540977 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:35.542790 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:35.544771 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:35.544777 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:35.544779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:35.545021 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:35.545250 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:35.545250 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:35.545250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:35.545356 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:35.545664 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:35.545668 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:35.545806 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 14:53:35.545893 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:53:35.546031 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
I20260928 14:53:35.546835 25383 heartbeater.cc:347] Connected to a master server at 127.20.116.190:45801
I20260928 14:53:35.546965 25383 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:35.547149 25383 heartbeater.cc:511] Master 127.20.116.190:45801 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.116.131@KRBTEST.COM; defaulting to no policy
I20260928 14:53:35.547586 25058 ts_manager.cc:194] Registered new tserver with Master: acfd31e4126247afaa206c441de6bcf7 (127.20.116.130:39237)
I20260928 14:53:35.547986 25058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:42769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
I20260928 14:53:35.565850 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:45801
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:35409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:35.648597 25388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:35.648814 25388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:35.648841 25388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:35.649971 25388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:35.650004 25388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:35.650030 25388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:53:35.651734 25388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:45801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:35.651950 25388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:35.652130 25388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:35.657291 25394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:35.657287 25393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:35.657338 25388 server_base.cc:1034] running on GCE node
W20260928 14:53:35.657291 25396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:35.657754 25388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:35.658030 25388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:35.659183 25388 hybrid_clock.cc:648] HybridClock initialized: now 1790607215659169 us; error 26 us; skew 500 ppm
Sep 28 14:53:35 dist-test-slave-pxl6 krb5kdc[25014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607215, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:35.661932 25388 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:53:35.662329 25388 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:35.662379 25388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:35.662473 25388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:35.663594 25388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "f6cd50d4ef43487f9db567a7c7a2d7f9"
format_stamp: "Formatted at 2026-09-28 14:53:35 on dist-test-slave-pxl6"
server_key: "a2e6a9c009565ea71279c72c453774a5"
server_key_iv: "a4e88b524ecc94fc004641c0de3f9a3d"
server_key_version: "encryptionkey@0"
I20260928 14:53:35.663913 25388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "f6cd50d4ef43487f9db567a7c7a2d7f9"
format_stamp: "Formatted at 2026-09-28 14:53:35 on dist-test-slave-pxl6"
server_key: "a2e6a9c009565ea71279c72c453774a5"
server_key_iv: "a4e88b524ecc94fc004641c0de3f9a3d"
server_key_version: "encryptionkey@0"
I20260928 14:53:35.665169 25388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:53:35.665910 25403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:35.666100 25388 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:53:35.666189 25388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "f6cd50d4ef43487f9db567a7c7a2d7f9"
format_stamp: "Formatted at 2026-09-28 14:53:35 on dist-test-slave-pxl6"
server_key: "a2e6a9c009565ea71279c72c453774a5"
server_key_iv: "a4e88b524ecc94fc004641c0de3f9a3d"
server_key_version: "encryptionkey@0"
I20260928 14:53:35.666244 25388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:35.677511 25388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:35.679924 25388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:35.680040 25388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:35.680239 25388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:35.680603 25388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:35.680644 25388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:35.680670 25388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:35.680683 25388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:35.686635 25388 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:44603
I20260928 14:53:35.686698 25516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:44603 every 8 connection(s)
I20260928 14:53:35.687004 25388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:53:35.691053 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 25388
I20260928 14:53:35.691149 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:53:35.691329 20946 external_mini_cluster.cc:1428] Setting key 88cc83ea237c748d3853ed066f1d5e8f
Sep 28 14:53:35 dist-test-slave-pxl6 krb5kdc[25014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607215, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:35.697453 25098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:35.687802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:56883 (local address 127.20.116.190:45801)
0928 14:53:35.687919 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:35.687922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:35.688106 (+ 184us) server_negotiation.cc:407] Connection header received
0928 14:53:35.688714 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:35.688715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:35.688730 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:35.688775 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:35.689751 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.690266 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.691781 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.692086 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.692143 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:35.694383 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:35.694384 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:35.694385 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:35.694392 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:35.695907 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:35.696328 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:35.696328 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:35.696329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:35.696382 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:35.696757 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:35.696759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:35.696759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:35.696894 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:35.696964 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:35.697297 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 14:53:35.697374 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:53:35.697654 25519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:35.687766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:45801 (local address 127.20.116.131:56883)
0928 14:53:35.687977 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:35.688015 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:35.688588 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:35.688865 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:35.688869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:35.689140 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:35.689649 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:35.689653 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.690365 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.690366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:35.691597 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:35.691600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.692167 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.692168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:35.692310 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:35.692886 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:35.692895 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:35.694296 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:35.696005 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:35.696012 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:35.696014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:35.696253 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:35.696467 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:35.696469 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:35.696469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:35.696608 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:35.697009 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:35.697016 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:35.697157 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 14:53:35.697262 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 14:53:35.697400 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":81,"threads_started":1}
I20260928 14:53:35.698225 25517 heartbeater.cc:347] Connected to a master server at 127.20.116.190:45801
I20260928 14:53:35.698328 25517 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:35.698459 25517 heartbeater.cc:511] Master 127.20.116.190:45801 requested a full tablet report, sending...
I20260928 14:53:35.698926 25058 ts_manager.cc:194] Registered new tserver with Master: f6cd50d4ef43487f9db567a7c7a2d7f9 (127.20.116.131:44603)
I20260928 14:53:35.699286 25058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:56883
I20260928 14:53:35.703239 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:53:35.711758 25098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:35.704604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40322 (local address 127.20.116.190:45801)
0928 14:53:35.704684 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:35.704687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:35.704803 (+ 116us) server_negotiation.cc:407] Connection header received
0928 14:53:35.704886 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:35.704887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:35.704898 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:35.704943 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:35.705534 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.706023 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.706832 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.707039 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.707125 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:35.708706 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:35.708707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:35.708708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:35.708715 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:35.710236 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:35.710697 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:35.710699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:35.710699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:35.710784 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:35.711007 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:35.711007 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:35.711008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:35.711275 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:35.711398 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:35.711523 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 14:53:35.711624 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:53:35.713860 25058 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:53:35.716065 25058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:53:35.723441 25533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:35.719301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39766 (local address 127.20.116.131:44603)
0928 14:53:35.719500 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:35.719504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:35.719700 (+ 196us) server_negotiation.cc:407] Connection header received
0928 14:53:35.719816 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:35.719818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:35.719835 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:35.719906 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:35.720106 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.720794 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.722163 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.722921 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.722997 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:35.723055 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:35.723312 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 14:53:35.723357 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":75,"threads_started":1}
I20260928 14:53:35.723459 25530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:35.718984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40026 (local address 127.20.116.130:39237)
0928 14:53:35.719137 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:35.719142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:35.719298 (+ 156us) server_negotiation.cc:407] Connection header received
0928 14:53:35.719407 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:35.719410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:35.719426 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:35.719524 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:35.719713 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.720898 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.722406 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.722974 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.723077 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:35.723135 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:35.723343 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 14:53:35.723383 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260928 14:53:35.723433 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:35.719115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:44603 (local address 127.0.0.1:39766)
0928 14:53:35.719577 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:35.719588 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:35.719641 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:35.719991 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:35.719992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:35.719994 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:35.720030 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:35.720031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.720899 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.720901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:35.722068 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:35.722070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.723095 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.723096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:35.723182 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:35.723196 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 14:53:35.723274 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:53:35.723337 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":73,"thread_start_us":64,"threads_started":1}
I20260928 14:53:35.723692 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:35.719012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35100 (local address 127.20.116.129:38107)
0928 14:53:35.719223 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:35.719227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:35.719324 (+ 97us) server_negotiation.cc:407] Connection header received
0928 14:53:35.719638 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:35.719641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:35.719668 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:35.719742 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:35.719958 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.721001 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.722383 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.723102 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.723169 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:35.723230 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:35.723475 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 14:53:35.723585 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
I20260928 14:53:35.723688 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:35.718883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:39237 (local address 127.0.0.1:40026)
0928 14:53:35.719190 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:35.719237 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:35.719317 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:35.719528 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:35.719531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:35.719534 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:35.719616 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:35.719618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.721026 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.721027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:35.722295 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:35.722297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.723077 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.723078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:35.723182 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:35.723199 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 14:53:35.723577 (+ 378us) client_negotiation.cc:240] Negotiation successful
0928 14:53:35.723629 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":62,"thread_start_us":73,"threads_started":1}
I20260928 14:53:35.723803 25529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:35.718890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:38107 (local address 127.0.0.1:35100)
0928 14:53:35.719191 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:35.719238 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:35.719317 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:35.719837 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:35.719838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:35.719841 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:35.719882 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:35.719884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.721099 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.721099 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:35.722295 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:35.722297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.723265 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.723266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:35.723320 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:35.723323 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:53:35.723369 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 14:53:35.723401 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":6,"thread_start_us":96,"threads_started":1}
I20260928 14:53:35.724891 25183 tablet_service.cc:1467] Processing CreateTablet for tablet e2d2c233e4a340a58a963b6c2533958c (DEFAULT_TABLE table=test-table [id=4fa5198330cb42f6ac2d628247027a33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:35.724893 25451 tablet_service.cc:1467] Processing CreateTablet for tablet e2d2c233e4a340a58a963b6c2533958c (DEFAULT_TABLE table=test-table [id=4fa5198330cb42f6ac2d628247027a33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:35.725273 25183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d2c233e4a340a58a963b6c2533958c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:35.725273 25451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d2c233e4a340a58a963b6c2533958c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:35.726943 25317 tablet_service.cc:1467] Processing CreateTablet for tablet e2d2c233e4a340a58a963b6c2533958c (DEFAULT_TABLE table=test-table [id=4fa5198330cb42f6ac2d628247027a33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:35.727288 25317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d2c233e4a340a58a963b6c2533958c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:35.728776 25534 tablet_bootstrap.cc:492] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9: Bootstrap starting.
I20260928 14:53:35.729013 25535 tablet_bootstrap.cc:492] T e2d2c233e4a340a58a963b6c2533958c P dc42141fe76f41aab49a033f121d13b0: Bootstrap starting.
I20260928 14:53:35.729737 25535 tablet_bootstrap.cc:654] T e2d2c233e4a340a58a963b6c2533958c P dc42141fe76f41aab49a033f121d13b0: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:35.730403 25535 log.cc:824] T e2d2c233e4a340a58a963b6c2533958c P dc42141fe76f41aab49a033f121d13b0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:35.730486 25536 tablet_bootstrap.cc:492] T e2d2c233e4a340a58a963b6c2533958c P acfd31e4126247afaa206c441de6bcf7: Bootstrap starting.
I20260928 14:53:35.731190 25536 tablet_bootstrap.cc:654] T e2d2c233e4a340a58a963b6c2533958c P acfd31e4126247afaa206c441de6bcf7: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:35.731297 25534 tablet_bootstrap.cc:654] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:35.731426 25535 tablet_bootstrap.cc:492] T e2d2c233e4a340a58a963b6c2533958c P dc42141fe76f41aab49a033f121d13b0: No bootstrap required, opened a new log
I20260928 14:53:35.731515 25535 ts_tablet_manager.cc:1397] T e2d2c233e4a340a58a963b6c2533958c P dc42141fe76f41aab49a033f121d13b0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:53:35.731851 25536 log.cc:824] T e2d2c233e4a340a58a963b6c2533958c P acfd31e4126247afaa206c441de6bcf7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:35.731949 25534 log.cc:824] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:35.732580 25536 tablet_bootstrap.cc:492] T e2d2c233e4a340a58a963b6c2533958c P acfd31e4126247afaa206c441de6bcf7: No bootstrap required, opened a new log
I20260928 14:53:35.732664 25536 ts_tablet_manager.cc:1397] T e2d2c233e4a340a58a963b6c2533958c P acfd31e4126247afaa206c441de6bcf7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:53:35.732738 25534 tablet_bootstrap.cc:492] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9: No bootstrap required, opened a new log
I20260928 14:53:35.732810 25534 ts_tablet_manager.cc:1397] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 14:53:35.733771 25535 raft_consensus.cc:348] T e2d2c233e4a340a58a963b6c2533958c P dc42141fe76f41aab49a033f121d13b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfd31e4126247afaa206c441de6bcf7" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 39237 } } peers { permanent_uuid: "dc42141fe76f41aab49a033f121d13b0" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38107 } } peers { permanent_uuid: "f6cd50d4ef43487f9db567a7c7a2d7f9" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 44603 } }
I20260928 14:53:35.733946 25535 raft_consensus.cc:374] T e2d2c233e4a340a58a963b6c2533958c P dc42141fe76f41aab49a033f121d13b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:35.734079 25535 raft_consensus.cc:729] T e2d2c233e4a340a58a963b6c2533958c P dc42141fe76f41aab49a033f121d13b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc42141fe76f41aab49a033f121d13b0, State: Initialized, Role: FOLLOWER
I20260928 14:53:35.734198 25535 consensus_queue.cc:260] T e2d2c233e4a340a58a963b6c2533958c P dc42141fe76f41aab49a033f121d13b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfd31e4126247afaa206c441de6bcf7" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 39237 } } peers { permanent_uuid: "dc42141fe76f41aab49a033f121d13b0" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38107 } } peers { permanent_uuid: "f6cd50d4ef43487f9db567a7c7a2d7f9" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 44603 } }
I20260928 14:53:35.734325 25536 raft_consensus.cc:348] T e2d2c233e4a340a58a963b6c2533958c P acfd31e4126247afaa206c441de6bcf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfd31e4126247afaa206c441de6bcf7" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 39237 } } peers { permanent_uuid: "dc42141fe76f41aab49a033f121d13b0" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38107 } } peers { permanent_uuid: "f6cd50d4ef43487f9db567a7c7a2d7f9" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 44603 } }
I20260928 14:53:35.734490 25536 raft_consensus.cc:374] T e2d2c233e4a340a58a963b6c2533958c P acfd31e4126247afaa206c441de6bcf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:35.734505 25535 ts_tablet_manager.cc:1428] T e2d2c233e4a340a58a963b6c2533958c P dc42141fe76f41aab49a033f121d13b0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:53:35.734557 25536 raft_consensus.cc:729] T e2d2c233e4a340a58a963b6c2533958c P acfd31e4126247afaa206c441de6bcf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acfd31e4126247afaa206c441de6bcf7, State: Initialized, Role: FOLLOWER
I20260928 14:53:35.734687 25536 consensus_queue.cc:260] T e2d2c233e4a340a58a963b6c2533958c P acfd31e4126247afaa206c441de6bcf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfd31e4126247afaa206c441de6bcf7" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 39237 } } peers { permanent_uuid: "dc42141fe76f41aab49a033f121d13b0" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38107 } } peers { permanent_uuid: "f6cd50d4ef43487f9db567a7c7a2d7f9" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 44603 } }
I20260928 14:53:35.734982 25536 ts_tablet_manager.cc:1428] T e2d2c233e4a340a58a963b6c2533958c P acfd31e4126247afaa206c441de6bcf7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:53:35.734918 25534 raft_consensus.cc:348] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfd31e4126247afaa206c441de6bcf7" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 39237 } } peers { permanent_uuid: "dc42141fe76f41aab49a033f121d13b0" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38107 } } peers { permanent_uuid: "f6cd50d4ef43487f9db567a7c7a2d7f9" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 44603 } }
I20260928 14:53:35.734994 25383 heartbeater.cc:503] Master 127.20.116.190:45801 was elected leader, sending a full tablet report...
I20260928 14:53:35.735076 25534 raft_consensus.cc:374] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:35.735111 25534 raft_consensus.cc:729] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6cd50d4ef43487f9db567a7c7a2d7f9, State: Initialized, Role: FOLLOWER
I20260928 14:53:35.735138 25249 heartbeater.cc:503] Master 127.20.116.190:45801 was elected leader, sending a full tablet report...
I20260928 14:53:35.735217 25534 consensus_queue.cc:260] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfd31e4126247afaa206c441de6bcf7" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 39237 } } peers { permanent_uuid: "dc42141fe76f41aab49a033f121d13b0" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38107 } } peers { permanent_uuid: "f6cd50d4ef43487f9db567a7c7a2d7f9" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 44603 } }
I20260928 14:53:35.735491 25534 ts_tablet_manager.cc:1428] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:53:35.735603 25517 heartbeater.cc:503] Master 127.20.116.190:45801 was elected leader, sending a full tablet report...
W20260928 14:53:35.785715 25384 tablet.cc:2369] T e2d2c233e4a340a58a963b6c2533958c P acfd31e4126247afaa206c441de6bcf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:53:35.810822 25542 raft_consensus.cc:482] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:53:35.810998 25542 raft_consensus.cc:504] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfd31e4126247afaa206c441de6bcf7" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 39237 } } peers { permanent_uuid: "dc42141fe76f41aab49a033f121d13b0" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38107 } } peers { permanent_uuid: "f6cd50d4ef43487f9db567a7c7a2d7f9" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 44603 } }
I20260928 14:53:35.811417 25542 leader_election.cc:288] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acfd31e4126247afaa206c441de6bcf7 (127.20.116.130:39237), dc42141fe76f41aab49a033f121d13b0 (127.20.116.129:38107)
I20260928 14:53:35.814356 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:35.811721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:48641 (local address 127.20.116.129:38107)
0928 14:53:35.811781 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:35.811784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:35.811798 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:53:35.811832 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:35.811833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:35.811844 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:35.811917 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:35.812090 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.812587 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.813586 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.813964 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.814073 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:35.814083 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:35.814240 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 14:53:35.814282 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 14:53:35.814356 25519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:35.811637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:38107 (local address 127.20.116.131:48641)
0928 14:53:35.811711 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:35.811724 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:35.811775 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:35.811924 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:35.811925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:35.811927 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:35.812005 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:35.812007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.812676 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.812677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:35.813469 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:35.813472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.814072 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.814073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:35.814160 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:35.814164 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:53:35.814232 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 14:53:35.814277 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 14:53:35.814465 25543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:35.811721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:39237 (local address 127.20.116.131:56349)
0928 14:53:35.811889 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:35.811898 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:35.811973 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:35.812114 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:35.812115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:35.812116 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:35.812196 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:35.812199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.812843 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.812844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:35.813581 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:35.813583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.814194 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.814195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:35.814272 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:35.814275 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:53:35.814340 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 14:53:35.814382 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":114,"threads_started":1}
I20260928 14:53:35.814502 25530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:35.811835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:56349 (local address 127.20.116.130:39237)
0928 14:53:35.811881 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:35.811884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:35.811968 (+ 84us) server_negotiation.cc:407] Connection header received
0928 14:53:35.812031 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:35.812032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:35.812042 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:35.812112 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:35.812272 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.812752 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.813698 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.814085 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.814187 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:35.814197 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:35.814360 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 14:53:35.814403 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:53:35.814817 25203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d2c233e4a340a58a963b6c2533958c" candidate_uuid: "f6cd50d4ef43487f9db567a7c7a2d7f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc42141fe76f41aab49a033f121d13b0" is_pre_election: true
I20260928 14:53:35.815038 25203 raft_consensus.cc:2454] T e2d2c233e4a340a58a963b6c2533958c P dc42141fe76f41aab49a033f121d13b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6cd50d4ef43487f9db567a7c7a2d7f9 in term 0.
I20260928 14:53:35.815078 25337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d2c233e4a340a58a963b6c2533958c" candidate_uuid: "f6cd50d4ef43487f9db567a7c7a2d7f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acfd31e4126247afaa206c441de6bcf7" is_pre_election: true
I20260928 14:53:35.815224 25337 raft_consensus.cc:2454] T e2d2c233e4a340a58a963b6c2533958c P acfd31e4126247afaa206c441de6bcf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6cd50d4ef43487f9db567a7c7a2d7f9 in term 0.
I20260928 14:53:35.815313 25405 leader_election.cc:302] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc42141fe76f41aab49a033f121d13b0, f6cd50d4ef43487f9db567a7c7a2d7f9; no voters:
I20260928 14:53:35.815445 25542 raft_consensus.cc:2781] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:53:35.815501 25542 raft_consensus.cc:482] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:53:35.815533 25542 raft_consensus.cc:3037] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:35.816398 25542 raft_consensus.cc:504] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfd31e4126247afaa206c441de6bcf7" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 39237 } } peers { permanent_uuid: "dc42141fe76f41aab49a033f121d13b0" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38107 } } peers { permanent_uuid: "f6cd50d4ef43487f9db567a7c7a2d7f9" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 44603 } }
I20260928 14:53:35.816572 25542 leader_election.cc:288] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9 [CANDIDATE]: Term 1 election: Requested vote from peers acfd31e4126247afaa206c441de6bcf7 (127.20.116.130:39237), dc42141fe76f41aab49a033f121d13b0 (127.20.116.129:38107)
I20260928 14:53:35.816749 25203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d2c233e4a340a58a963b6c2533958c" candidate_uuid: "f6cd50d4ef43487f9db567a7c7a2d7f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc42141fe76f41aab49a033f121d13b0"
I20260928 14:53:35.816839 25337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d2c233e4a340a58a963b6c2533958c" candidate_uuid: "f6cd50d4ef43487f9db567a7c7a2d7f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acfd31e4126247afaa206c441de6bcf7"
I20260928 14:53:35.816845 25203 raft_consensus.cc:3037] T e2d2c233e4a340a58a963b6c2533958c P dc42141fe76f41aab49a033f121d13b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:35.816938 25337 raft_consensus.cc:3037] T e2d2c233e4a340a58a963b6c2533958c P acfd31e4126247afaa206c441de6bcf7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:35.817710 25203 raft_consensus.cc:2454] T e2d2c233e4a340a58a963b6c2533958c P dc42141fe76f41aab49a033f121d13b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6cd50d4ef43487f9db567a7c7a2d7f9 in term 1.
I20260928 14:53:35.817783 25337 raft_consensus.cc:2454] T e2d2c233e4a340a58a963b6c2533958c P acfd31e4126247afaa206c441de6bcf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6cd50d4ef43487f9db567a7c7a2d7f9 in term 1.
I20260928 14:53:35.817955 25407 leader_election.cc:302] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acfd31e4126247afaa206c441de6bcf7, f6cd50d4ef43487f9db567a7c7a2d7f9; no voters:
I20260928 14:53:35.818055 25542 raft_consensus.cc:2781] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:35.818220 25542 raft_consensus.cc:686] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9 [term 1 LEADER]: Becoming Leader. State: Replica: f6cd50d4ef43487f9db567a7c7a2d7f9, State: Running, Role: LEADER
I20260928 14:53:35.818351 25542 consensus_queue.cc:237] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfd31e4126247afaa206c441de6bcf7" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 39237 } } peers { permanent_uuid: "dc42141fe76f41aab49a033f121d13b0" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38107 } } peers { permanent_uuid: "f6cd50d4ef43487f9db567a7c7a2d7f9" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 44603 } }
I20260928 14:53:35.819324 25057 catalog_manager.cc:5495] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6cd50d4ef43487f9db567a7c7a2d7f9 (127.20.116.131). New cstate: current_term: 1 leader_uuid: "f6cd50d4ef43487f9db567a7c7a2d7f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfd31e4126247afaa206c441de6bcf7" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 39237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc42141fe76f41aab49a033f121d13b0" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6cd50d4ef43487f9db567a7c7a2d7f9" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 44603 } health_report { overall_health: HEALTHY } } }
I20260928 14:53:35.844084 25533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:35.841443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39778 (local address 127.20.116.131:44603)
0928 14:53:35.841510 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:35.841513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:35.841523 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:53:35.841605 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:35.841606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:35.841625 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:35.841707 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:35.841887 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.842373 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.843254 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:35.843454 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:35.843542 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:35.843708 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:35.843824 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:35.843923 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:35.843943 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 14:53:35.843965 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:53:35.846155 25203 raft_consensus.cc:1261] T e2d2c233e4a340a58a963b6c2533958c P dc42141fe76f41aab49a033f121d13b0 [term 1 FOLLOWER]: Refusing update from remote peer f6cd50d4ef43487f9db567a7c7a2d7f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:35.846153 25337 raft_consensus.cc:1261] T e2d2c233e4a340a58a963b6c2533958c P acfd31e4126247afaa206c441de6bcf7 [term 1 FOLLOWER]: Refusing update from remote peer f6cd50d4ef43487f9db567a7c7a2d7f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:35.846575 25544 consensus_queue.cc:1035] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acfd31e4126247afaa206c441de6bcf7" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 39237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:35.846727 25542 consensus_queue.cc:1035] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc42141fe76f41aab49a033f121d13b0" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:35.849881 25547 mvcc.cc:205] Tried to move back new op lower bound from 7334327156102914048 to 7334327155992920064. Current Snapshot: MvccSnapshot[applied={T|T < 7334327156102914048}]
I20260928 14:53:35.850165 25549 mvcc.cc:205] Tried to move back new op lower bound from 7334327156102914048 to 7334327155992920064. Current Snapshot: MvccSnapshot[applied={T|T < 7334327156102914048}]
I20260928 14:53:35.850484 25548 mvcc.cc:205] Tried to move back new op lower bound from 7334327156102914048 to 7334327155992920064. Current Snapshot: MvccSnapshot[applied={T|T < 7334327156102914048}]
I20260928 14:53:35.853077 25057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:35.853205 25057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:35.854090 25057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47c7bb19ff9c40bca24d04cd6db2369c: Sending DeleteTablet for 3 replicas of tablet e2d2c233e4a340a58a963b6c2533958c
I20260928 14:53:35.854642 25183 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d2c233e4a340a58a963b6c2533958c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:35 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:35100
I20260928 14:53:35.854718 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 25120
I20260928 14:53:35.855208 25451 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d2c233e4a340a58a963b6c2533958c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:35 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:39766
I20260928 14:53:35.855715 25317 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d2c233e4a340a58a963b6c2533958c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:35 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:40026
I20260928 14:53:35.856278 25558 tablet_replica.cc:330] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9: stopping tablet replica
I20260928 14:53:35.856381 25558 raft_consensus.cc:2229] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:53:35.856528 25558 raft_consensus.cc:2258] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:35.857012 25558 ts_tablet_manager.cc:1905] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:35.858682 25559 tablet_replica.cc:330] T e2d2c233e4a340a58a963b6c2533958c P acfd31e4126247afaa206c441de6bcf7: stopping tablet replica
I20260928 14:53:35.858817 25559 raft_consensus.cc:2229] T e2d2c233e4a340a58a963b6c2533958c P acfd31e4126247afaa206c441de6bcf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:35.858945 25559 raft_consensus.cc:2258] T e2d2c233e4a340a58a963b6c2533958c P acfd31e4126247afaa206c441de6bcf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:35.859043 25558 ts_tablet_manager.cc:1918] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:53:35.859123 25558 log.cc:1196] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/e2d2c233e4a340a58a963b6c2533958c
I20260928 14:53:35.859473 25558 ts_tablet_manager.cc:1939] T e2d2c233e4a340a58a963b6c2533958c P f6cd50d4ef43487f9db567a7c7a2d7f9: Deleting consensus metadata
I20260928 14:53:35.859699 25559 ts_tablet_manager.cc:1905] T e2d2c233e4a340a58a963b6c2533958c P acfd31e4126247afaa206c441de6bcf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:35.859942 25043 catalog_manager.cc:4841] TS f6cd50d4ef43487f9db567a7c7a2d7f9 (127.20.116.131:44603): tablet e2d2c233e4a340a58a963b6c2533958c (table test-table [id=4fa5198330cb42f6ac2d628247027a33]) successfully deleted
I20260928 14:53:35.861644 25559 ts_tablet_manager.cc:1918] T e2d2c233e4a340a58a963b6c2533958c P acfd31e4126247afaa206c441de6bcf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:53:35.861742 25559 log.cc:1196] T e2d2c233e4a340a58a963b6c2533958c P acfd31e4126247afaa206c441de6bcf7: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/e2d2c233e4a340a58a963b6c2533958c
I20260928 14:53:35.862051 25559 ts_tablet_manager.cc:1939] T e2d2c233e4a340a58a963b6c2533958c P acfd31e4126247afaa206c441de6bcf7: Deleting consensus metadata
I20260928 14:53:35.862519 25045 catalog_manager.cc:4841] TS acfd31e4126247afaa206c441de6bcf7 (127.20.116.130:39237): tablet e2d2c233e4a340a58a963b6c2533958c (table test-table [id=4fa5198330cb42f6ac2d628247027a33]) successfully deleted
W20260928 14:53:35.862593 25043 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.116.129:38107 (error 108)
I20260928 14:53:35.862936 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 25254
I20260928 14:53:35.863613 25529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:35.863034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:38107 (local address 127.0.0.1:35116)
0928 14:53:35.863503 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:35.863563 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.129:38107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444}
W20260928 14:53:35.863766 25043 catalog_manager.cc:4584] TS dc42141fe76f41aab49a033f121d13b0 (127.20.116.129:38107): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e2d2c233e4a340a58a963b6c2533958c: Network error: Client connection negotiation failed: client connection to 127.20.116.129:38107: connect: Connection refused (error 111)
I20260928 14:53:35.869205 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 25388
I20260928 14:53:35.874348 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 25027
2026-09-28T14:53:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2024 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T14:53:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:53:35Z Disabled control of system clock
I20260928 14:53:35.903741 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:35853
--webserver_interface=127.20.116.190
--webserver_port=0
--builtin_ntp_servers=127.20.116.148:33705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:35853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 14:53:35.987890 25564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:35.988126 25564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:35.988150 25564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:35.989408 25564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:53:35.989455 25564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:35.989467 25564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:35.989480 25564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:53:35.989492 25564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:53:35.990914 25564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:35853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:35853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:35.991143 25564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:35.991408 25564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:35.996822 25572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:35.996809 25569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:35.996812 25570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:35.996923 25564 server_base.cc:1034] running on GCE node
I20260928 14:53:35.997421 25564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:35.997752 25564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:35.998960 25564 hybrid_clock.cc:648] HybridClock initialized: now 1790607215998936 us; error 60 us; skew 500 ppm
I20260928 14:53:36.000491 25564 webserver.cc:460] Webserver started at http://127.20.116.190:44841/ using document root <none> and password file <none>
I20260928 14:53:36.000758 25564 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:36.000834 25564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:36.000996 25564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:36.002117 25564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "d524d61acd3242cfb6b8a7785aaa0bf1"
format_stamp: "Formatted at 2026-09-28 14:53:35 on dist-test-slave-pxl6"
server_key: "bacc0cdadc592bcf8d54da9b4bbf2264"
server_key_iv: "6494bd47a31f44da71803797e6e81bb7"
server_key_version: "encryptionkey@0"
I20260928 14:53:36.002493 25564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "d524d61acd3242cfb6b8a7785aaa0bf1"
format_stamp: "Formatted at 2026-09-28 14:53:35 on dist-test-slave-pxl6"
server_key: "bacc0cdadc592bcf8d54da9b4bbf2264"
server_key_iv: "6494bd47a31f44da71803797e6e81bb7"
server_key_version: "encryptionkey@0"
I20260928 14:53:36.003957 25564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:36.004770 25578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:36.004963 25564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:36.005075 25564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "d524d61acd3242cfb6b8a7785aaa0bf1"
format_stamp: "Formatted at 2026-09-28 14:53:35 on dist-test-slave-pxl6"
server_key: "bacc0cdadc592bcf8d54da9b4bbf2264"
server_key_iv: "6494bd47a31f44da71803797e6e81bb7"
server_key_version: "encryptionkey@0"
I20260928 14:53:36.005167 25564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:36.008731 25564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:36.009025 25564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:36.009174 25564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:36.013496 25564 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:35853
I20260928 14:53:36.013589 25630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:35853 every 8 connection(s)
I20260928 14:53:36.013916 25564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:53:36.014708 25631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:36.017573 25631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1: Bootstrap starting.
I20260928 14:53:36.018162 25631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:36.018384 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 25564
I20260928 14:53:36.018481 25631 log.cc:824] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:36.018587 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:53:36.018780 20946 external_mini_cluster.cc:1428] Setting key 90e626f0f67301e5a77ef0b16195084e
I20260928 14:53:36.019338 25631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1: No bootstrap required, opened a new log
I20260928 14:53:36.021051 25631 raft_consensus.cc:348] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d524d61acd3242cfb6b8a7785aaa0bf1" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35853 } }
I20260928 14:53:36.021195 25631 raft_consensus.cc:374] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:36.021217 25631 raft_consensus.cc:729] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d524d61acd3242cfb6b8a7785aaa0bf1, State: Initialized, Role: FOLLOWER
I20260928 14:53:36.021353 25631 consensus_queue.cc:260] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d524d61acd3242cfb6b8a7785aaa0bf1" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35853 } }
I20260928 14:53:36.021446 25631 raft_consensus.cc:388] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:53:36.021482 25631 raft_consensus.cc:482] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:53:36.021560 25631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:36.022145 25631 raft_consensus.cc:504] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d524d61acd3242cfb6b8a7785aaa0bf1" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35853 } }
I20260928 14:53:36.022277 25631 leader_election.cc:302] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d524d61acd3242cfb6b8a7785aaa0bf1; no voters:
I20260928 14:53:36.022475 25631 leader_election.cc:288] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:53:36.022504 25637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:36.022711 25637 raft_consensus.cc:686] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1 [term 1 LEADER]: Becoming Leader. State: Replica: d524d61acd3242cfb6b8a7785aaa0bf1, State: Running, Role: LEADER
I20260928 14:53:36.022812 25631 sys_catalog.cc:564] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:36.022848 25637 consensus_queue.cc:237] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d524d61acd3242cfb6b8a7785aaa0bf1" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35853 } }
I20260928 14:53:36.023037 25636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:36.019352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58664 (local address 127.20.116.190:35853)
0928 14:53:36.019558 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:36.019568 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:36.019603 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:53:36.021060 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:36.021068 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:36.021126 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:36.021321 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:36.022129 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:36.022132 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:36.022133 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:53:36.022136 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:36.022190 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:53:36.022200 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:36.022281 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:36.022359 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 14:53:36.022424 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260928 14:53:36.023288 25638 sys_catalog.cc:455] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d524d61acd3242cfb6b8a7785aaa0bf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d524d61acd3242cfb6b8a7785aaa0bf1" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35853 } } }
I20260928 14:53:36.023283 25639 sys_catalog.cc:455] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d524d61acd3242cfb6b8a7785aaa0bf1. Latest consensus state: current_term: 1 leader_uuid: "d524d61acd3242cfb6b8a7785aaa0bf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d524d61acd3242cfb6b8a7785aaa0bf1" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35853 } } }
I20260928 14:53:36.023383 25638 sys_catalog.cc:458] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:36.023387 25639 sys_catalog.cc:458] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:36.023861 25642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:53:36.024539 25642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:53:36.026588 25642 catalog_manager.cc:1347] Generated new cluster ID: 80f4a2fcac854e4f8753a625a6fdbd52
I20260928 14:53:36.026680 25642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:53:36.048295 25642 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:53:36.048527 25642 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:53:36.051002 25642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1: Generated new TSK 0
I20260928 14:53:36.051229 25642 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:53:36.059890 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35853
--builtin_ntp_servers=127.20.116.148:33705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 14:53:36.148594 25656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:36.148866 25656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:36.148908 25656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:36.150247 25656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:36.150328 25656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:36.150375 25656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:53:36.151871 25656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:36.152185 25656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:36.152516 25656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:36.158017 25662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:36.158012 25661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:36.158242 25656 server_base.cc:1034] running on GCE node
W20260928 14:53:36.158331 25664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:36.158629 25656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:36.158921 25656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:36.160120 25656 hybrid_clock.cc:648] HybridClock initialized: now 1790607216160100 us; error 35 us; skew 500 ppm
I20260928 14:53:36.161454 25656 webserver.cc:460] Webserver started at http://127.20.116.129:34569/ using document root <none> and password file <none>
I20260928 14:53:36.161710 25656 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:36.161782 25656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:36.161921 25656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:36.163129 25656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "2b9e76a78f9f4e6183f32be8b514e245"
format_stamp: "Formatted at 2026-09-28 14:53:36 on dist-test-slave-pxl6"
server_key: "ecbeccd7089663941418e7e3733a8fb9"
server_key_iv: "9a406f27a495dcf63f113068cc2217ad"
server_key_version: "encryptionkey@0"
I20260928 14:53:36.163497 25656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "2b9e76a78f9f4e6183f32be8b514e245"
format_stamp: "Formatted at 2026-09-28 14:53:36 on dist-test-slave-pxl6"
server_key: "ecbeccd7089663941418e7e3733a8fb9"
server_key_iv: "9a406f27a495dcf63f113068cc2217ad"
server_key_version: "encryptionkey@0"
I20260928 14:53:36.164875 25656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:53:36.165719 25670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:36.165911 25656 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:36.166039 25656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "2b9e76a78f9f4e6183f32be8b514e245"
format_stamp: "Formatted at 2026-09-28 14:53:36 on dist-test-slave-pxl6"
server_key: "ecbeccd7089663941418e7e3733a8fb9"
server_key_iv: "9a406f27a495dcf63f113068cc2217ad"
server_key_version: "encryptionkey@0"
I20260928 14:53:36.166121 25656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:36.169677 25656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:36.169973 25656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:36.170122 25656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:36.170339 25656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:36.170722 25656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:36.170775 25656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:36.170815 25656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:36.170841 25656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:36.176838 25656 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:41299
I20260928 14:53:36.176889 25784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:41299 every 8 connection(s)
I20260928 14:53:36.177220 25656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:53:36.180642 25636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:36.178143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:47467 (local address 127.20.116.190:35853)
0928 14:53:36.178212 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:36.178214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:36.178459 (+ 245us) server_negotiation.cc:407] Connection header received
0928 14:53:36.179057 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:36.179058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:36.179066 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:36.179134 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:36.180404 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:36.180406 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:36.180406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:53:36.180411 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:36.180429 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:53:36.180433 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:36.180500 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:36.180567 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 14:53:36.180582 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:53:36.180855 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:36.178107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35853 (local address 127.20.116.129:47467)
0928 14:53:36.178322 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:36.178364 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:36.178913 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:36.179215 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:36.179219 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:36.179227 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:36.180238 (+ 1011us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:53:36.180239 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:36.180268 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:53:36.180298 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:36.180498 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:36.180501 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:36.180503 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:53:36.180556 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 14:53:36.180618 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":114,"threads_started":1}
I20260928 14:53:36.181437 25785 heartbeater.cc:347] Connected to a master server at 127.20.116.190:35853
I20260928 14:53:36.181541 25785 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:36.181762 25785 heartbeater.cc:511] Master 127.20.116.190:35853 requested a full tablet report, sending...
I20260928 14:53:36.182446 25595 ts_manager.cc:194] Registered new tserver with Master: 2b9e76a78f9f4e6183f32be8b514e245 (127.20.116.129:41299)
I20260928 14:53:36.185437 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 25656
I20260928 14:53:36.185530 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:53:36.185835 20946 external_mini_cluster.cc:1428] Setting key c694e6fd22bc49be3e32cdc95910a593
I20260928 14:53:36.186872 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35853
--builtin_ntp_servers=127.20.116.148:33705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 14:53:36.270973 25788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:36.271206 25788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:36.271268 25788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:36.272471 25788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:36.272516 25788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:36.272540 25788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:53:36.273872 25788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:36.274084 25788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:36.274353 25788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:36.279487 25794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:36.279513 25793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:36.279505 25796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:36.279700 25788 server_base.cc:1034] running on GCE node
I20260928 14:53:36.279901 25788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:36.280105 25788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:36.281250 25788 hybrid_clock.cc:648] HybridClock initialized: now 1790607216281225 us; error 36 us; skew 500 ppm
I20260928 14:53:36.282661 25788 webserver.cc:460] Webserver started at http://127.20.116.130:33743/ using document root <none> and password file <none>
I20260928 14:53:36.282915 25788 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:36.282965 25788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:36.283128 25788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:36.284268 25788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "7532e000722d4d3abc2061e4fd073fee"
format_stamp: "Formatted at 2026-09-28 14:53:36 on dist-test-slave-pxl6"
server_key: "d594676a5c3bd1f782d68e8a0af5b694"
server_key_iv: "7553fba3196c01da655a295017ff5667"
server_key_version: "encryptionkey@0"
I20260928 14:53:36.284642 25788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "7532e000722d4d3abc2061e4fd073fee"
format_stamp: "Formatted at 2026-09-28 14:53:36 on dist-test-slave-pxl6"
server_key: "d594676a5c3bd1f782d68e8a0af5b694"
server_key_iv: "7553fba3196c01da655a295017ff5667"
server_key_version: "encryptionkey@0"
I20260928 14:53:36.286026 25788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:53:36.286906 25802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:36.287077 25788 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:36.287205 25788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "7532e000722d4d3abc2061e4fd073fee"
format_stamp: "Formatted at 2026-09-28 14:53:36 on dist-test-slave-pxl6"
server_key: "d594676a5c3bd1f782d68e8a0af5b694"
server_key_iv: "7553fba3196c01da655a295017ff5667"
server_key_version: "encryptionkey@0"
I20260928 14:53:36.287288 25788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:36.290889 25788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:36.291215 25788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:36.291378 25788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:36.291602 25788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:36.291945 25788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:36.291999 25788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:36.292040 25788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:36.292068 25788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:36.298194 25788 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:35095
I20260928 14:53:36.298246 25916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:35095 every 8 connection(s)
I20260928 14:53:36.298749 25788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:53:36.301853 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 25788
I20260928 14:53:36.301964 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:53:36.302167 25636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:36.299680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:55315 (local address 127.20.116.190:35853)
0928 14:53:36.299753 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:36.299756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:36.300014 (+ 258us) server_negotiation.cc:407] Connection header received
0928 14:53:36.300503 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:36.300504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:36.300548 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:36.300606 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:36.301811 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:36.301813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:36.301813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:53:36.301819 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:36.301838 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:53:36.301843 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:36.301922 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:36.302077 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 14:53:36.302094 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:53:36.302183 20946 external_mini_cluster.cc:1428] Setting key ffbe4d407611fbdda8fca4a020df9cbe
I20260928 14:53:36.302436 25919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:36.299679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35853 (local address 127.20.116.130:55315)
0928 14:53:36.299895 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:36.299932 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:36.300412 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:36.300662 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:36.300665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:36.300674 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:36.301650 (+ 976us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:53:36.301650 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:36.301682 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:53:36.301711 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:36.301976 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:36.301982 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:36.301985 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:53:36.302085 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 14:53:36.302165 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":99,"threads_started":1}
I20260928 14:53:36.303030 25917 heartbeater.cc:347] Connected to a master server at 127.20.116.190:35853
I20260928 14:53:36.303130 25917 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:36.303294 25917 heartbeater.cc:511] Master 127.20.116.190:35853 requested a full tablet report, sending...
I20260928 14:53:36.303336 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35853
--builtin_ntp_servers=127.20.116.148:33705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 14:53:36.303683 25594 ts_manager.cc:194] Registered new tserver with Master: 7532e000722d4d3abc2061e4fd073fee (127.20.116.130:35095)
W20260928 14:53:36.387248 25920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:36.387462 25920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:36.387486 25920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:36.388703 25920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:36.388743 25920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:36.388768 25920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:53:36.390177 25920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:36.390380 25920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:36.390663 25920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:36.395783 25925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:36.395813 25928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:36.395805 25926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:36.395924 25920 server_base.cc:1034] running on GCE node
I20260928 14:53:36.396202 25920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:36.396463 25920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:36.397632 25920 hybrid_clock.cc:648] HybridClock initialized: now 1790607216397609 us; error 42 us; skew 500 ppm
I20260928 14:53:36.399457 25920 webserver.cc:460] Webserver started at http://127.20.116.131:42221/ using document root <none> and password file <none>
I20260928 14:53:36.399730 25920 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:36.399803 25920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:36.399950 25920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:36.401532 25920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "a4d6399c4079435bb38ca4f1c05f230c"
format_stamp: "Formatted at 2026-09-28 14:53:36 on dist-test-slave-pxl6"
server_key: "e3de655a5228c4803d0bea90f7ecee44"
server_key_iv: "3b0ddf9a946d5f9a88464db65c2c68be"
server_key_version: "encryptionkey@0"
I20260928 14:53:36.402065 25920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "a4d6399c4079435bb38ca4f1c05f230c"
format_stamp: "Formatted at 2026-09-28 14:53:36 on dist-test-slave-pxl6"
server_key: "e3de655a5228c4803d0bea90f7ecee44"
server_key_iv: "3b0ddf9a946d5f9a88464db65c2c68be"
server_key_version: "encryptionkey@0"
I20260928 14:53:36.404510 25920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 14:53:36.405807 25935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:36.406009 25920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:36.406122 25920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "a4d6399c4079435bb38ca4f1c05f230c"
format_stamp: "Formatted at 2026-09-28 14:53:36 on dist-test-slave-pxl6"
server_key: "e3de655a5228c4803d0bea90f7ecee44"
server_key_iv: "3b0ddf9a946d5f9a88464db65c2c68be"
server_key_version: "encryptionkey@0"
I20260928 14:53:36.406222 25920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:36.411510 25920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:36.411861 25920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:36.412017 25920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:36.412297 25920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:36.412676 25920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:36.412729 25920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:36.412776 25920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:36.412798 25920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:36.420589 25920 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:33419
I20260928 14:53:36.420616 26048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:33419 every 8 connection(s)
I20260928 14:53:36.421001 25920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:53:36.425148 25636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:36.422223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:49435 (local address 127.20.116.190:35853)
0928 14:53:36.422271 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:36.422273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:36.422284 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:53:36.423112 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:36.423114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:36.423121 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:36.423217 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:36.424855 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:36.424857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:36.424857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:53:36.424872 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:36.424889 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:53:36.424894 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:36.424952 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:36.425060 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 14:53:36.425074 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:53:36.425387 26051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:36.421852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35853 (local address 127.20.116.131:49435)
0928 14:53:36.422087 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:36.422155 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:36.422968 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:36.423300 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:36.423305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:36.423317 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:36.424665 (+ 1348us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:53:36.424666 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:36.424709 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:53:36.424749 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:36.424967 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:36.424972 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:36.424976 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:53:36.425063 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:53:36.425131 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
I20260928 14:53:36.425920 26049 heartbeater.cc:347] Connected to a master server at 127.20.116.190:35853
I20260928 14:53:36.426028 26049 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:36.426214 26049 heartbeater.cc:511] Master 127.20.116.190:35853 requested a full tablet report, sending...
I20260928 14:53:36.426687 25594 ts_manager.cc:194] Registered new tserver with Master: a4d6399c4079435bb38ca4f1c05f230c (127.20.116.131:33419)
I20260928 14:53:36.428412 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 25920
I20260928 14:53:36.428540 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:53:36.428745 20946 external_mini_cluster.cc:1428] Setting key c9f44f707802eeaa1721c0baddc6c46e
I20260928 14:53:36.429836 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:53:36.432932 25636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:36.431516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58678 (local address 127.20.116.190:35853)
0928 14:53:36.431574 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:36.431576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:36.431749 (+ 173us) server_negotiation.cc:407] Connection header received
0928 14:53:36.431836 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:36.431837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:36.431844 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:36.431895 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:36.432676 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:36.432678 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:36.432678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:53:36.432682 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:36.432697 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:53:36.432701 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:36.432763 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:36.432838 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 14:53:36.432854 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 14:53:36.435006 25594 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:53:36.437297 25594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:53:36.441608 26057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:36.439851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:33419 (local address 127.0.0.1:45130)
0928 14:53:36.439991 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:36.440032 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:36.440093 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:36.440436 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:36.440443 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:36.440451 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:36.441169 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:53:36.441170 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:36.441186 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:53:36.441212 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:36.441425 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:36.441430 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:36.441433 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:53:36.441519 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 14:53:36.441533 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"spinlock_wait_cycles":896,"thread_start_us":96,"threads_started":1}
I20260928 14:53:36.441622 26061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:36.440132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.20.116.131:33419)
0928 14:53:36.440299 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:36.440304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:36.440316 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:53:36.440343 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:36.440345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:36.440356 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:36.440434 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:36.441298 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:36.441305 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:36.441306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:53:36.441310 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:36.441345 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:53:36.441373 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:36.441432 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:36.441530 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 14:53:36.441548 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260928 14:53:36.442104 26059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:36.439853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:35095 (local address 127.0.0.1:37966)
0928 14:53:36.440320 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:36.440333 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:36.440382 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:36.440956 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:36.440956 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:36.440959 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:36.441723 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:53:36.441724 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:36.441730 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:53:36.441745 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:36.441982 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:36.441983 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:36.441984 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 14:53:36.442028 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 14:53:36.442041 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":84,"threads_started":1}
I20260928 14:53:36.442232 26058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:36.439852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:41299 (local address 127.0.0.1:37912)
0928 14:53:36.440458 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:36.440467 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:36.440502 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:36.440955 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:36.440956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:36.440961 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:36.441840 (+ 879us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:53:36.441840 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:36.441845 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:53:36.441854 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:36.442083 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:36.442084 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:36.442085 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 14:53:36.442125 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 14:53:36.442168 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":4,"thread_start_us":107,"threads_started":1}
I20260928 14:53:36.442340 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:36.440122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37966 (local address 127.20.116.130:35095)
0928 14:53:36.440825 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:36.440829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:36.440838 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:53:36.440864 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:36.440867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:36.440875 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:36.440952 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:36.441816 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:36.441827 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:36.441827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:53:36.441832 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:36.441868 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:53:36.441896 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:36.441954 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:36.442233 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 14:53:36.442251 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":91,"threads_started":1}
I20260928 14:53:36.442509 26062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:36.440127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37912 (local address 127.20.116.129:41299)
0928 14:53:36.440378 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:36.440384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:36.440556 (+ 172us) server_negotiation.cc:407] Connection header received
0928 14:53:36.440588 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:36.440590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:36.440601 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:36.440669 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:36.441929 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:36.441939 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:36.441939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:53:36.441944 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:36.441978 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:53:36.442006 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:36.442059 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:36.442401 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 14:53:36.442418 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":80,"threads_started":1}
I20260928 14:53:36.442457 25983 tablet_service.cc:1467] Processing CreateTablet for tablet c94074eb492a4ac286a0b9fc8b0ddb15 (DEFAULT_TABLE table=test-table [id=f9dd9b312970459280a83478a266b9f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:36.442919 25983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94074eb492a4ac286a0b9fc8b0ddb15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:36.443145 25850 tablet_service.cc:1467] Processing CreateTablet for tablet c94074eb492a4ac286a0b9fc8b0ddb15 (DEFAULT_TABLE table=test-table [id=f9dd9b312970459280a83478a266b9f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:36.443470 25850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94074eb492a4ac286a0b9fc8b0ddb15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:36.445652 26063 tablet_bootstrap.cc:492] T c94074eb492a4ac286a0b9fc8b0ddb15 P a4d6399c4079435bb38ca4f1c05f230c: Bootstrap starting.
I20260928 14:53:36.446094 25718 tablet_service.cc:1467] Processing CreateTablet for tablet c94074eb492a4ac286a0b9fc8b0ddb15 (DEFAULT_TABLE table=test-table [id=f9dd9b312970459280a83478a266b9f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:36.446343 26063 tablet_bootstrap.cc:654] T c94074eb492a4ac286a0b9fc8b0ddb15 P a4d6399c4079435bb38ca4f1c05f230c: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:36.446444 25718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94074eb492a4ac286a0b9fc8b0ddb15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:36.446704 26063 log.cc:824] T c94074eb492a4ac286a0b9fc8b0ddb15 P a4d6399c4079435bb38ca4f1c05f230c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:36.447522 26063 tablet_bootstrap.cc:492] T c94074eb492a4ac286a0b9fc8b0ddb15 P a4d6399c4079435bb38ca4f1c05f230c: No bootstrap required, opened a new log
I20260928 14:53:36.447598 26063 ts_tablet_manager.cc:1397] T c94074eb492a4ac286a0b9fc8b0ddb15 P a4d6399c4079435bb38ca4f1c05f230c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 14:53:36.449894 26066 tablet_bootstrap.cc:492] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245: Bootstrap starting.
I20260928 14:53:36.450366 26063 raft_consensus.cc:348] T c94074eb492a4ac286a0b9fc8b0ddb15 P a4d6399c4079435bb38ca4f1c05f230c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532e000722d4d3abc2061e4fd073fee" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35095 } } peers { permanent_uuid: "a4d6399c4079435bb38ca4f1c05f230c" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 33419 } } peers { permanent_uuid: "2b9e76a78f9f4e6183f32be8b514e245" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41299 } }
I20260928 14:53:36.450618 26063 raft_consensus.cc:374] T c94074eb492a4ac286a0b9fc8b0ddb15 P a4d6399c4079435bb38ca4f1c05f230c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:36.450667 26063 raft_consensus.cc:729] T c94074eb492a4ac286a0b9fc8b0ddb15 P a4d6399c4079435bb38ca4f1c05f230c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d6399c4079435bb38ca4f1c05f230c, State: Initialized, Role: FOLLOWER
I20260928 14:53:36.450717 26064 tablet_bootstrap.cc:492] T c94074eb492a4ac286a0b9fc8b0ddb15 P 7532e000722d4d3abc2061e4fd073fee: Bootstrap starting.
I20260928 14:53:36.450810 26063 consensus_queue.cc:260] T c94074eb492a4ac286a0b9fc8b0ddb15 P a4d6399c4079435bb38ca4f1c05f230c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532e000722d4d3abc2061e4fd073fee" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35095 } } peers { permanent_uuid: "a4d6399c4079435bb38ca4f1c05f230c" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 33419 } } peers { permanent_uuid: "2b9e76a78f9f4e6183f32be8b514e245" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41299 } }
I20260928 14:53:36.451124 26049 heartbeater.cc:503] Master 127.20.116.190:35853 was elected leader, sending a full tablet report...
I20260928 14:53:36.451107 26063 ts_tablet_manager.cc:1428] T c94074eb492a4ac286a0b9fc8b0ddb15 P a4d6399c4079435bb38ca4f1c05f230c: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260928 14:53:36.451375 26064 tablet_bootstrap.cc:654] T c94074eb492a4ac286a0b9fc8b0ddb15 P 7532e000722d4d3abc2061e4fd073fee: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:36.451066 26066 tablet_bootstrap.cc:654] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:36.451721 26064 log.cc:824] T c94074eb492a4ac286a0b9fc8b0ddb15 P 7532e000722d4d3abc2061e4fd073fee: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:36.451959 26066 log.cc:824] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:36.452659 26064 tablet_bootstrap.cc:492] T c94074eb492a4ac286a0b9fc8b0ddb15 P 7532e000722d4d3abc2061e4fd073fee: No bootstrap required, opened a new log
I20260928 14:53:36.452737 26064 ts_tablet_manager.cc:1397] T c94074eb492a4ac286a0b9fc8b0ddb15 P 7532e000722d4d3abc2061e4fd073fee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:53:36.454710 26064 raft_consensus.cc:348] T c94074eb492a4ac286a0b9fc8b0ddb15 P 7532e000722d4d3abc2061e4fd073fee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532e000722d4d3abc2061e4fd073fee" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35095 } } peers { permanent_uuid: "a4d6399c4079435bb38ca4f1c05f230c" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 33419 } } peers { permanent_uuid: "2b9e76a78f9f4e6183f32be8b514e245" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41299 } }
I20260928 14:53:36.454888 26064 raft_consensus.cc:374] T c94074eb492a4ac286a0b9fc8b0ddb15 P 7532e000722d4d3abc2061e4fd073fee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:36.454932 26064 raft_consensus.cc:729] T c94074eb492a4ac286a0b9fc8b0ddb15 P 7532e000722d4d3abc2061e4fd073fee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7532e000722d4d3abc2061e4fd073fee, State: Initialized, Role: FOLLOWER
I20260928 14:53:36.455062 26064 consensus_queue.cc:260] T c94074eb492a4ac286a0b9fc8b0ddb15 P 7532e000722d4d3abc2061e4fd073fee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532e000722d4d3abc2061e4fd073fee" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35095 } } peers { permanent_uuid: "a4d6399c4079435bb38ca4f1c05f230c" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 33419 } } peers { permanent_uuid: "2b9e76a78f9f4e6183f32be8b514e245" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41299 } }
I20260928 14:53:36.455366 26064 ts_tablet_manager.cc:1428] T c94074eb492a4ac286a0b9fc8b0ddb15 P 7532e000722d4d3abc2061e4fd073fee: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:53:36.455451 25917 heartbeater.cc:503] Master 127.20.116.190:35853 was elected leader, sending a full tablet report...
I20260928 14:53:36.455705 26066 tablet_bootstrap.cc:492] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245: No bootstrap required, opened a new log
I20260928 14:53:36.455788 26066 ts_tablet_manager.cc:1397] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 14:53:36.458043 26066 raft_consensus.cc:348] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532e000722d4d3abc2061e4fd073fee" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35095 } } peers { permanent_uuid: "a4d6399c4079435bb38ca4f1c05f230c" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 33419 } } peers { permanent_uuid: "2b9e76a78f9f4e6183f32be8b514e245" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41299 } }
I20260928 14:53:36.458202 26066 raft_consensus.cc:374] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:36.458226 26066 raft_consensus.cc:729] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b9e76a78f9f4e6183f32be8b514e245, State: Initialized, Role: FOLLOWER
I20260928 14:53:36.458369 26066 consensus_queue.cc:260] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532e000722d4d3abc2061e4fd073fee" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35095 } } peers { permanent_uuid: "a4d6399c4079435bb38ca4f1c05f230c" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 33419 } } peers { permanent_uuid: "2b9e76a78f9f4e6183f32be8b514e245" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41299 } }
I20260928 14:53:36.458676 26066 ts_tablet_manager.cc:1428] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 14:53:36.458704 25785 heartbeater.cc:503] Master 127.20.116.190:35853 was elected leader, sending a full tablet report...
W20260928 14:53:36.549335 25918 tablet.cc:2369] T c94074eb492a4ac286a0b9fc8b0ddb15 P 7532e000722d4d3abc2061e4fd073fee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:53:36.658341 26072 raft_consensus.cc:482] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:53:36.658536 26072 raft_consensus.cc:504] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532e000722d4d3abc2061e4fd073fee" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35095 } } peers { permanent_uuid: "a4d6399c4079435bb38ca4f1c05f230c" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 33419 } } peers { permanent_uuid: "2b9e76a78f9f4e6183f32be8b514e245" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41299 } }
I20260928 14:53:36.658996 26072 leader_election.cc:288] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7532e000722d4d3abc2061e4fd073fee (127.20.116.130:35095), a4d6399c4079435bb38ca4f1c05f230c (127.20.116.131:33419)
I20260928 14:53:36.660534 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:36.659222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:33419 (local address 127.20.116.129:38551)
0928 14:53:36.659284 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:36.659297 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:36.659340 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:36.659535 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:36.659537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:36.659541 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:36.660089 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:53:36.660090 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:36.660099 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:53:36.660115 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:36.660351 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:36.660352 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:36.660353 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 14:53:36.660431 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:53:36.660446 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 14:53:36.660578 26061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:36.659310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:38551 (local address 127.20.116.131:33419)
0928 14:53:36.659370 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:36.659372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:36.659384 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:53:36.659417 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:36.659418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:36.659424 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:36.659505 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:36.660223 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:36.660225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:36.660225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:53:36.660230 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:36.660269 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:53:36.660274 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:36.660342 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:36.660438 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 14:53:36.660467 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 14:53:36.660655 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:36.659293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:40411 (local address 127.20.116.130:35095)
0928 14:53:36.659373 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:36.659375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:36.659530 (+ 155us) server_negotiation.cc:407] Connection header received
0928 14:53:36.659556 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:36.659557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:36.659563 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:36.659617 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:36.660421 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:36.660423 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:36.660423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:53:36.660428 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:36.660446 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:53:36.660450 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:36.660506 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:36.660568 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 14:53:36.660583 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:53:36.660704 26073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:36.659247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:35095 (local address 127.20.116.129:40411)
0928 14:53:36.659424 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:36.659434 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:36.659485 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:36.659637 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:36.659638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:36.659673 (+ 35us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:36.660313 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:53:36.660314 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:36.660323 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:53:36.660336 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:36.660516 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:36.660517 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:36.660518 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 14:53:36.660570 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 14:53:36.660584 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":108,"threads_started":1}
I20260928 14:53:36.660882 26003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94074eb492a4ac286a0b9fc8b0ddb15" candidate_uuid: "2b9e76a78f9f4e6183f32be8b514e245" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4d6399c4079435bb38ca4f1c05f230c" is_pre_election: true
I20260928 14:53:36.661019 26003 raft_consensus.cc:2454] T c94074eb492a4ac286a0b9fc8b0ddb15 P a4d6399c4079435bb38ca4f1c05f230c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b9e76a78f9f4e6183f32be8b514e245 in term 0.
I20260928 14:53:36.661252 25871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94074eb492a4ac286a0b9fc8b0ddb15" candidate_uuid: "2b9e76a78f9f4e6183f32be8b514e245" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7532e000722d4d3abc2061e4fd073fee" is_pre_election: true
I20260928 14:53:36.661303 25673 leader_election.cc:302] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b9e76a78f9f4e6183f32be8b514e245, a4d6399c4079435bb38ca4f1c05f230c; no voters:
I20260928 14:53:36.661391 25871 raft_consensus.cc:2454] T c94074eb492a4ac286a0b9fc8b0ddb15 P 7532e000722d4d3abc2061e4fd073fee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b9e76a78f9f4e6183f32be8b514e245 in term 0.
I20260928 14:53:36.661459 26072 raft_consensus.cc:2781] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:53:36.661526 26072 raft_consensus.cc:482] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:53:36.661554 26072 raft_consensus.cc:3037] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:36.662288 26072 raft_consensus.cc:504] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532e000722d4d3abc2061e4fd073fee" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35095 } } peers { permanent_uuid: "a4d6399c4079435bb38ca4f1c05f230c" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 33419 } } peers { permanent_uuid: "2b9e76a78f9f4e6183f32be8b514e245" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41299 } }
I20260928 14:53:36.662437 26072 leader_election.cc:288] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245 [CANDIDATE]: Term 1 election: Requested vote from peers 7532e000722d4d3abc2061e4fd073fee (127.20.116.130:35095), a4d6399c4079435bb38ca4f1c05f230c (127.20.116.131:33419)
I20260928 14:53:36.662636 26003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94074eb492a4ac286a0b9fc8b0ddb15" candidate_uuid: "2b9e76a78f9f4e6183f32be8b514e245" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4d6399c4079435bb38ca4f1c05f230c"
I20260928 14:53:36.662644 25871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94074eb492a4ac286a0b9fc8b0ddb15" candidate_uuid: "2b9e76a78f9f4e6183f32be8b514e245" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7532e000722d4d3abc2061e4fd073fee"
I20260928 14:53:36.662740 26003 raft_consensus.cc:3037] T c94074eb492a4ac286a0b9fc8b0ddb15 P a4d6399c4079435bb38ca4f1c05f230c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:36.662742 25871 raft_consensus.cc:3037] T c94074eb492a4ac286a0b9fc8b0ddb15 P 7532e000722d4d3abc2061e4fd073fee [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:36.663451 26003 raft_consensus.cc:2454] T c94074eb492a4ac286a0b9fc8b0ddb15 P a4d6399c4079435bb38ca4f1c05f230c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b9e76a78f9f4e6183f32be8b514e245 in term 1.
I20260928 14:53:36.663533 25871 raft_consensus.cc:2454] T c94074eb492a4ac286a0b9fc8b0ddb15 P 7532e000722d4d3abc2061e4fd073fee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b9e76a78f9f4e6183f32be8b514e245 in term 1.
I20260928 14:53:36.663668 25673 leader_election.cc:302] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b9e76a78f9f4e6183f32be8b514e245, a4d6399c4079435bb38ca4f1c05f230c; no voters:
I20260928 14:53:36.663753 26072 raft_consensus.cc:2781] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:36.663923 26072 raft_consensus.cc:686] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245 [term 1 LEADER]: Becoming Leader. State: Replica: 2b9e76a78f9f4e6183f32be8b514e245, State: Running, Role: LEADER
I20260928 14:53:36.664022 26072 consensus_queue.cc:237] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532e000722d4d3abc2061e4fd073fee" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35095 } } peers { permanent_uuid: "a4d6399c4079435bb38ca4f1c05f230c" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 33419 } } peers { permanent_uuid: "2b9e76a78f9f4e6183f32be8b514e245" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41299 } }
I20260928 14:53:36.664858 25594 catalog_manager.cc:5495] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b9e76a78f9f4e6183f32be8b514e245 (127.20.116.129). New cstate: current_term: 1 leader_uuid: "2b9e76a78f9f4e6183f32be8b514e245" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532e000722d4d3abc2061e4fd073fee" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4d6399c4079435bb38ca4f1c05f230c" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 33419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b9e76a78f9f4e6183f32be8b514e245" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41299 } health_report { overall_health: HEALTHY } } }
W20260928 14:53:36.671638 26050 tablet.cc:2369] T c94074eb492a4ac286a0b9fc8b0ddb15 P a4d6399c4079435bb38ca4f1c05f230c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:53:36.677366 26062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:36.676401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37916 (local address 127.20.116.129:41299)
0928 14:53:36.676450 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:36.676452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:36.676469 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:53:36.676541 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:36.676542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:36.676558 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:36.676607 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:36.677204 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:36.677205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:36.677205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:53:36.677208 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:36.677220 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:53:36.677224 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:36.677267 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:36.677301 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 14:53:36.677312 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 14:53:36.677825 25786 tablet.cc:2369] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:53:36.679342 25871 raft_consensus.cc:1261] T c94074eb492a4ac286a0b9fc8b0ddb15 P 7532e000722d4d3abc2061e4fd073fee [term 1 FOLLOWER]: Refusing update from remote peer 2b9e76a78f9f4e6183f32be8b514e245: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:36.679342 26003 raft_consensus.cc:1261] T c94074eb492a4ac286a0b9fc8b0ddb15 P a4d6399c4079435bb38ca4f1c05f230c [term 1 FOLLOWER]: Refusing update from remote peer 2b9e76a78f9f4e6183f32be8b514e245: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:36.679733 26074 consensus_queue.cc:1035] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4d6399c4079435bb38ca4f1c05f230c" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 33419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:36.679916 26072 consensus_queue.cc:1035] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7532e000722d4d3abc2061e4fd073fee" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:36.682305 26078 mvcc.cc:205] Tried to move back new op lower bound from 7334327159515357184 to 7334327159456452608. Current Snapshot: MvccSnapshot[applied={T|T < 7334327159515357184}]
I20260928 14:53:36.682521 26079 mvcc.cc:205] Tried to move back new op lower bound from 7334327159515357184 to 7334327159456452608. Current Snapshot: MvccSnapshot[applied={T|T < 7334327159515357184}]
I20260928 14:53:36.682958 26077 mvcc.cc:205] Tried to move back new op lower bound from 7334327159515357184 to 7334327159456452608. Current Snapshot: MvccSnapshot[applied={T|T < 7334327159515357184 or (T in {7334327159515357184})}]
I20260928 14:53:36.684738 25594 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:36.684842 25594 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:36.685626 25594 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d524d61acd3242cfb6b8a7785aaa0bf1: Sending DeleteTablet for 3 replicas of tablet c94074eb492a4ac286a0b9fc8b0ddb15
I20260928 14:53:36.685966 25850 tablet_service.cc:1514] Processing DeleteTablet for tablet c94074eb492a4ac286a0b9fc8b0ddb15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:36 UTC) from {username='slave'} at 127.0.0.1:37966
I20260928 14:53:36.686022 25983 tablet_service.cc:1514] Processing DeleteTablet for tablet c94074eb492a4ac286a0b9fc8b0ddb15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:36 UTC) from {username='slave'} at 127.0.0.1:45130
I20260928 14:53:36.686062 25718 tablet_service.cc:1514] Processing DeleteTablet for tablet c94074eb492a4ac286a0b9fc8b0ddb15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:36 UTC) from {username='slave'} at 127.0.0.1:37912
I20260928 14:53:36.686262 26089 tablet_replica.cc:330] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245: stopping tablet replica
I20260928 14:53:36.686273 26087 tablet_replica.cc:330] T c94074eb492a4ac286a0b9fc8b0ddb15 P 7532e000722d4d3abc2061e4fd073fee: stopping tablet replica
I20260928 14:53:36.686355 26089 raft_consensus.cc:2229] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:53:36.686372 26087 raft_consensus.cc:2229] T c94074eb492a4ac286a0b9fc8b0ddb15 P 7532e000722d4d3abc2061e4fd073fee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:36.686470 26087 raft_consensus.cc:2258] T c94074eb492a4ac286a0b9fc8b0ddb15 P 7532e000722d4d3abc2061e4fd073fee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:36.686479 26089 raft_consensus.cc:2258] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:36.686702 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 25656
I20260928 14:53:36.686887 26089 ts_tablet_manager.cc:1905] T c94074eb492a4ac286a0b9fc8b0ddb15 P 2b9e76a78f9f4e6183f32be8b514e245: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:36.687479 26087 ts_tablet_manager.cc:1905] T c94074eb492a4ac286a0b9fc8b0ddb15 P 7532e000722d4d3abc2061e4fd073fee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:36.689487 26087 ts_tablet_manager.cc:1918] T c94074eb492a4ac286a0b9fc8b0ddb15 P 7532e000722d4d3abc2061e4fd073fee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:53:36.689564 26087 log.cc:1196] T c94074eb492a4ac286a0b9fc8b0ddb15 P 7532e000722d4d3abc2061e4fd073fee: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/c94074eb492a4ac286a0b9fc8b0ddb15
I20260928 14:53:36.689854 26087 ts_tablet_manager.cc:1939] T c94074eb492a4ac286a0b9fc8b0ddb15 P 7532e000722d4d3abc2061e4fd073fee: Deleting consensus metadata
I20260928 14:53:36.690280 25582 catalog_manager.cc:4841] TS 7532e000722d4d3abc2061e4fd073fee (127.20.116.130:35095): tablet c94074eb492a4ac286a0b9fc8b0ddb15 (table test-table [id=f9dd9b312970459280a83478a266b9f2]) successfully deleted
I20260928 14:53:36.690666 26088 tablet_replica.cc:330] T c94074eb492a4ac286a0b9fc8b0ddb15 P a4d6399c4079435bb38ca4f1c05f230c: stopping tablet replica
I20260928 14:53:36.690794 26088 raft_consensus.cc:2229] T c94074eb492a4ac286a0b9fc8b0ddb15 P a4d6399c4079435bb38ca4f1c05f230c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:36.690930 26088 raft_consensus.cc:2258] T c94074eb492a4ac286a0b9fc8b0ddb15 P a4d6399c4079435bb38ca4f1c05f230c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:36.691329 26088 ts_tablet_manager.cc:1905] T c94074eb492a4ac286a0b9fc8b0ddb15 P a4d6399c4079435bb38ca4f1c05f230c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:36.692606 26088 ts_tablet_manager.cc:1918] T c94074eb492a4ac286a0b9fc8b0ddb15 P a4d6399c4079435bb38ca4f1c05f230c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:53:36.692696 26088 log.cc:1196] T c94074eb492a4ac286a0b9fc8b0ddb15 P a4d6399c4079435bb38ca4f1c05f230c: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/c94074eb492a4ac286a0b9fc8b0ddb15
I20260928 14:53:36.692961 26088 ts_tablet_manager.cc:1939] T c94074eb492a4ac286a0b9fc8b0ddb15 P a4d6399c4079435bb38ca4f1c05f230c: Deleting consensus metadata
I20260928 14:53:36.693418 25581 catalog_manager.cc:4841] TS a4d6399c4079435bb38ca4f1c05f230c (127.20.116.131:33419): tablet c94074eb492a4ac286a0b9fc8b0ddb15 (table test-table [id=f9dd9b312970459280a83478a266b9f2]) successfully deleted
W20260928 14:53:36.694311 25579 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.116.129:41299 (error 108)
I20260928 14:53:36.694479 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 25788
I20260928 14:53:36.694854 26058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:36.694735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:41299 (local address 127.0.0.1:37926)
0928 14:53:36.694779 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:36.694816 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.129:41299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 14:53:36.694959 25579 catalog_manager.cc:4584] TS 2b9e76a78f9f4e6183f32be8b514e245 (127.20.116.129:41299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c94074eb492a4ac286a0b9fc8b0ddb15: Network error: Client connection negotiation failed: client connection to 127.20.116.129:41299: connect: Connection refused (error 111)
I20260928 14:53:36.700543 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 25920
I20260928 14:53:36.705559 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 25564
2026-09-28T14:53:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (830 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 14:53:36.725915 20946 mini_oidc.cc:152] Starting JWKS server
I20260928 14:53:36.727130 20946 webserver.cc:460] Webserver started at https://127.0.0.1:33577/ using document root <none> and password file <none>
I20260928 14:53:36.727216 20946 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 14:53:36.727346 20946 webserver.cc:460] Webserver started at http://127.0.0.1:45109/ using document root <none> and password file <none>
2026-09-28T14:53:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:53:36Z Disabled control of system clock
I20260928 14:53:36.735715 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:43045
--webserver_interface=127.20.116.190
--webserver_port=0
--builtin_ntp_servers=127.20.116.148:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:43045
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33577/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607192794321-20946-0/testchainca.cert with env {}
W20260928 14:53:36.818213 26096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:36.818498 26096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:36.818539 26096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:36.819869 26096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:53:36.819955 26096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:36.819976 26096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:36.819998 26096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:53:36.820030 26096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:53:36.820070 26096 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:53:36.820089 26096 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33577/jwks.json
I20260928 14:53:36.821818 26096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:43045
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:43045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33577/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607192794321-20946-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:36.822160 26096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:36.822409 26096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:36.827587 26104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:36.827623 26102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:36.827625 26096 server_base.cc:1034] running on GCE node
W20260928 14:53:36.827595 26101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:36.828150 26096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:36.828467 26096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:36.829670 26096 hybrid_clock.cc:648] HybridClock initialized: now 1790607216829663 us; error 37 us; skew 500 ppm
I20260928 14:53:36.831012 26096 webserver.cc:460] Webserver started at http://127.20.116.190:34287/ using document root <none> and password file <none>
I20260928 14:53:36.831303 26096 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:36.831374 26096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:36.831535 26096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:36.832687 26096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "244dfab3d0cb489abceee8bf0e3862d0"
format_stamp: "Formatted at 2026-09-28 14:53:36 on dist-test-slave-pxl6"
server_key: "3da938dc9af239a854dfe9a9f8e1e092"
server_key_iv: "c15a3fd695d404179a2d122c79b18b56"
server_key_version: "encryptionkey@0"
I20260928 14:53:36.833082 26096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "244dfab3d0cb489abceee8bf0e3862d0"
format_stamp: "Formatted at 2026-09-28 14:53:36 on dist-test-slave-pxl6"
server_key: "3da938dc9af239a854dfe9a9f8e1e092"
server_key_iv: "c15a3fd695d404179a2d122c79b18b56"
server_key_version: "encryptionkey@0"
I20260928 14:53:36.834741 26096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:53:36.835700 26110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:36.835894 26096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:36.836006 26096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "244dfab3d0cb489abceee8bf0e3862d0"
format_stamp: "Formatted at 2026-09-28 14:53:36 on dist-test-slave-pxl6"
server_key: "3da938dc9af239a854dfe9a9f8e1e092"
server_key_iv: "c15a3fd695d404179a2d122c79b18b56"
server_key_version: "encryptionkey@0"
I20260928 14:53:36.836103 26096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:36.849961 26096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:36.850325 26096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:36.850485 26096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:36.854620 26096 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:43045
I20260928 14:53:36.854764 26162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:43045 every 8 connection(s)
I20260928 14:53:36.855099 26096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:53:36.856081 26163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:36.858372 26163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0: Bootstrap starting.
I20260928 14:53:36.859042 26163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:36.859300 26163 log.cc:824] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:36.859983 26163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0: No bootstrap required, opened a new log
I20260928 14:53:36.861893 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 26096
I20260928 14:53:36.861729 26163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244dfab3d0cb489abceee8bf0e3862d0" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43045 } }
I20260928 14:53:36.861958 26163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:36.862002 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:53:36.862002 26163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244dfab3d0cb489abceee8bf0e3862d0, State: Initialized, Role: FOLLOWER
I20260928 14:53:36.862206 20946 external_mini_cluster.cc:1428] Setting key 178312f6b0d813827ef5c383d2cbcab8
I20260928 14:53:36.862174 26163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244dfab3d0cb489abceee8bf0e3862d0" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43045 } }
I20260928 14:53:36.862259 26163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:53:36.862293 26163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:53:36.862349 26163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:36.863312 26163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244dfab3d0cb489abceee8bf0e3862d0" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43045 } }
I20260928 14:53:36.863447 26163 leader_election.cc:302] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 244dfab3d0cb489abceee8bf0e3862d0; no voters:
I20260928 14:53:36.863675 26163 leader_election.cc:288] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:53:36.863705 26168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:36.863811 26168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0 [term 1 LEADER]: Becoming Leader. State: Replica: 244dfab3d0cb489abceee8bf0e3862d0, State: Running, Role: LEADER
I20260928 14:53:36.863895 26168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244dfab3d0cb489abceee8bf0e3862d0" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43045 } }
I20260928 14:53:36.863967 26163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:36.864269 26168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "244dfab3d0cb489abceee8bf0e3862d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244dfab3d0cb489abceee8bf0e3862d0" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43045 } } }
I20260928 14:53:36.864341 26168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:36.864456 26168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 244dfab3d0cb489abceee8bf0e3862d0. Latest consensus state: current_term: 1 leader_uuid: "244dfab3d0cb489abceee8bf0e3862d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244dfab3d0cb489abceee8bf0e3862d0" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43045 } } }
I20260928 14:53:36.864527 26168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:36.864665 26173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:53:36.865227 26173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:53:36.867017 26173 catalog_manager.cc:1347] Generated new cluster ID: 0c1e962a4a1f4feeabd5de195ccc1021
I20260928 14:53:36.867071 26173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:53:36.868759 26167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:36.862730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38772 (local address 127.20.116.190:43045)
0928 14:53:36.862975 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:36.862985 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:36.863023 (+ 38us) server_negotiation.cc:407] Connection header received
0928 14:53:36.863133 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:36.863139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:36.863209 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:36.863350 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:36.863574 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:36.865367 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:36.866466 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:36.867132 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:36.867260 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:36.868016 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:36.868020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:36.868021 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:53:36.868213 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:36.868247 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:53:36.868258 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:36.868337 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:36.868415 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 14:53:36.868592 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":98,"threads_started":1}
I20260928 14:53:36.869765 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:53:36.889777 26173 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:53:36.890456 26173 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:53:36.894721 26173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 244dfab3d0cb489abceee8bf0e3862d0: Generated new TSK 0
I20260928 14:53:36.894958 26173 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 14:53:36.927489 26193 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 14:53:36.927641 26193 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260928 14:53:36.928169 26192 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:36.904206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43045 (local address 127.0.0.1:38784)
0928 14:53:36.904422 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:36.904436 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:36.904514 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:36.904770 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:36.904771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:36.904774 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:53:36.904843 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:36.904845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:36.905477 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:36.905496 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:36.906490 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:36.906493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:36.906937 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:36.906939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:36.907009 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:36.907036 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:53:36.927751 (+ 20715us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33577/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 14:53:36.927875 (+ 124us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.116.190:43045: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33577/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":92,"threads_started":1}
I20260928 14:53:36.928602 26167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:36.904357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38784 (local address 127.20.116.190:43045)
0928 14:53:36.904424 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:36.904426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:36.904489 (+ 63us) server_negotiation.cc:407] Connection header received
0928 14:53:36.904665 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:36.904666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:36.904677 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:36.904738 (+ 61us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:53:36.904911 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:36.905396 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:36.906653 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:36.906842 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:36.906900 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:36.907179 (+ 279us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:53:36.927551 (+ 20372us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33577/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 14:53:36.928484 (+ 933us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38784: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33577/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":41}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.20.116.190:43045: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33577/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
W20260928 14:53:36.928777 26167 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38784: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33577/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
I20260928 14:53:36.928790 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 26096
2026-09-28T14:53:36Z chronyd exiting
I20260928 14:53:36.965286 20946 test_util.cc:175] -----------------------------------------------
I20260928 14:53:36.965370 20946 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607192794321-20946-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (240 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 14:53:36.966216 20946 mini_oidc.cc:152] Starting JWKS server
I20260928 14:53:36.967032 20946 webserver.cc:460] Webserver started at https://127.0.0.1:36301/ using document root <none> and password file <none>
I20260928 14:53:36.967078 20946 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 14:53:36.967245 20946 webserver.cc:460] Webserver started at http://127.0.0.1:42467/ using document root <none> and password file <none>
2026-09-28T14:53:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:53:36Z Disabled control of system clock
I20260928 14:53:36.975749 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:43485
--webserver_interface=127.20.116.190
--webserver_port=0
--builtin_ntp_servers=127.20.116.148:38053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:43485
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36301/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607192794321-20946-0/testchainca.cert with env {}
W20260928 14:53:37.060225 26199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:37.060460 26199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:37.060488 26199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:37.061798 26199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:53:37.061842 26199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:37.061856 26199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:37.061875 26199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:53:37.061890 26199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:53:37.061903 26199 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:53:37.061916 26199 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36301/jwks.json
I20260928 14:53:37.063673 26199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:38053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:43485
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:43485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36301/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607192794321-20946-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:37.063898 26199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:37.064077 26199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:37.069464 26205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:37.069525 26199 server_base.cc:1034] running on GCE node
W20260928 14:53:37.069482 26207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:37.069481 26204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:37.069989 26199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:37.070325 26199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:37.071513 26199 hybrid_clock.cc:648] HybridClock initialized: now 1790607217071483 us; error 66 us; skew 500 ppm
I20260928 14:53:37.072959 26199 webserver.cc:460] Webserver started at http://127.20.116.190:34823/ using document root <none> and password file <none>
I20260928 14:53:37.073266 26199 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:37.073346 26199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:37.073500 26199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:37.074713 26199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "240a3da675be40b3a91549abb7c20865"
format_stamp: "Formatted at 2026-09-28 14:53:37 on dist-test-slave-pxl6"
server_key: "1c75b68917dd827f783d054886e26d46"
server_key_iv: "d1d46e02be7d01b1704e309ad2a971af"
server_key_version: "encryptionkey@0"
I20260928 14:53:37.075112 26199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "240a3da675be40b3a91549abb7c20865"
format_stamp: "Formatted at 2026-09-28 14:53:37 on dist-test-slave-pxl6"
server_key: "1c75b68917dd827f783d054886e26d46"
server_key_iv: "d1d46e02be7d01b1704e309ad2a971af"
server_key_version: "encryptionkey@0"
I20260928 14:53:37.076508 26199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:53:37.077400 26213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:37.077598 26199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:53:37.077710 26199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "240a3da675be40b3a91549abb7c20865"
format_stamp: "Formatted at 2026-09-28 14:53:37 on dist-test-slave-pxl6"
server_key: "1c75b68917dd827f783d054886e26d46"
server_key_iv: "d1d46e02be7d01b1704e309ad2a971af"
server_key_version: "encryptionkey@0"
I20260928 14:53:37.077796 26199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:37.095000 26199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:37.095377 26199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:37.095552 26199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:37.099876 26199 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:43485
I20260928 14:53:37.099929 26266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:43485 every 8 connection(s)
I20260928 14:53:37.100410 26199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:53:37.101112 26267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:37.104102 26267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865: Bootstrap starting.
I20260928 14:53:37.104743 26267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:37.105096 26267 log.cc:824] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:37.105939 26267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865: No bootstrap required, opened a new log
I20260928 14:53:37.107752 26267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240a3da675be40b3a91549abb7c20865" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43485 } }
I20260928 14:53:37.107905 26267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:37.107928 26267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 240a3da675be40b3a91549abb7c20865, State: Initialized, Role: FOLLOWER
I20260928 14:53:37.108095 26267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240a3da675be40b3a91549abb7c20865" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43485 } }
I20260928 14:53:37.108173 26267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:53:37.108197 26267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:53:37.108248 26267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:37.108860 26267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240a3da675be40b3a91549abb7c20865" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43485 } }
I20260928 14:53:37.108954 26267 leader_election.cc:302] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 240a3da675be40b3a91549abb7c20865; no voters:
I20260928 14:53:37.109148 26267 leader_election.cc:288] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:53:37.109278 26270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:37.109467 26270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865 [term 1 LEADER]: Becoming Leader. State: Replica: 240a3da675be40b3a91549abb7c20865, State: Running, Role: LEADER
I20260928 14:53:37.109477 26267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:37.109622 26270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240a3da675be40b3a91549abb7c20865" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43485 } }
I20260928 14:53:37.110047 26271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "240a3da675be40b3a91549abb7c20865" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240a3da675be40b3a91549abb7c20865" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43485 } } }
I20260928 14:53:37.110205 26271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865 [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:37.110291 26272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 240a3da675be40b3a91549abb7c20865. Latest consensus state: current_term: 1 leader_uuid: "240a3da675be40b3a91549abb7c20865" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240a3da675be40b3a91549abb7c20865" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43485 } } }
I20260928 14:53:37.110381 26272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865 [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:37.110773 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 26199
I20260928 14:53:37.110808 26279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:53:37.110875 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:53:37.111084 20946 external_mini_cluster.cc:1428] Setting key 365f9ca33df7a85552172f62acc8476c
I20260928 14:53:37.111474 26279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:53:37.113646 26279 catalog_manager.cc:1347] Generated new cluster ID: c9aaef1fd3c7459488c686c1081283b3
I20260928 14:53:37.113730 26279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:53:37.120217 26289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:37.111686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53140 (local address 127.20.116.190:43485)
0928 14:53:37.114695 (+ 3009us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:37.114706 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:37.114744 (+ 38us) server_negotiation.cc:407] Connection header received
0928 14:53:37.114865 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:37.114872 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:37.114942 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:37.115096 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:37.115332 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:37.116795 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:37.117885 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:37.118581 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:37.118717 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:37.119384 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:37.119388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:37.119389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:53:37.119598 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:37.119644 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:53:37.119668 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:37.119739 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:37.119812 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 14:53:37.119996 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2914,"thread_start_us":85,"threads_started":1}
I20260928 14:53:37.121200 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:53:37.122846 26279 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:53:37.123371 26279 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:53:37.129567 26279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 240a3da675be40b3a91549abb7c20865: Generated new TSK 0
I20260928 14:53:37.129813 26279 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 14:53:37.157171 26297 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 14:53:37.157267 26297 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 14:53:37.157696 26289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:37.135234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53148 (local address 127.20.116.190:43485)
0928 14:53:37.135284 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:37.135287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:37.135386 (+ 99us) server_negotiation.cc:407] Connection header received
0928 14:53:37.135468 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:37.135469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:37.135482 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:37.135583 (+ 101us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:53:37.135733 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:37.136461 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:37.137312 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:37.137555 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:37.137645 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:37.137839 (+ 194us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:53:37.157394 (+ 19555us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36301/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 14:53:37.157613 (+ 219us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53148: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36301/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 14:53:37.157774 26289 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53148: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36301/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 14:53:37.157783 26296 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:37.135173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43485 (local address 127.0.0.1:53148)
0928 14:53:37.135318 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:37.135327 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:37.135390 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:37.135582 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:37.135583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:37.135584 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:53:37.135649 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:37.135652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:37.136592 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:37.136593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:37.137219 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:37.137221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:37.137650 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:37.137652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:37.137715 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:37.137721 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:53:37.157549 (+ 19828us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36301/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 14:53:37.157672 (+ 123us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.116.190:43485: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36301/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260928 14:53:37.158070 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 26199
2026-09-28T14:53:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (230 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 14:53:37.196554 20946 mini_oidc.cc:152] Starting JWKS server
I20260928 14:53:37.197270 20946 webserver.cc:460] Webserver started at https://127.0.0.1:37091/ using document root <none> and password file <none>
I20260928 14:53:37.197315 20946 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 14:53:37.197463 20946 webserver.cc:460] Webserver started at http://127.0.0.1:34385/ using document root <none> and password file <none>
2026-09-28T14:53:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:53:37Z Disabled control of system clock
I20260928 14:53:37.205696 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:36061
--webserver_interface=127.20.116.190
--webserver_port=0
--builtin_ntp_servers=127.20.116.148:33213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:36061
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37091/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 14:53:37.289089 26303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:37.289324 26303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:37.289347 26303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:37.290525 26303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:53:37.290602 26303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:37.290616 26303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:37.290628 26303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:53:37.290640 26303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:53:37.290653 26303 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:53:37.290720 26303 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37091/jwks.json
I20260928 14:53:37.292232 26303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:36061
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:36061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37091/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:37.292528 26303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:37.292855 26303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:37.298188 26311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:37.298188 26309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:37.298348 26303 server_base.cc:1034] running on GCE node
W20260928 14:53:37.298484 26308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:37.298810 26303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:37.299145 26303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:37.300302 26303 hybrid_clock.cc:648] HybridClock initialized: now 1790607217300287 us; error 36 us; skew 500 ppm
I20260928 14:53:37.301808 26303 webserver.cc:460] Webserver started at http://127.20.116.190:36901/ using document root <none> and password file <none>
I20260928 14:53:37.302068 26303 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:37.302119 26303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:37.302255 26303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:37.303467 26303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "231e92c575c64d6f8300a88f2af39828"
format_stamp: "Formatted at 2026-09-28 14:53:37 on dist-test-slave-pxl6"
server_key: "e5873ac6cdd74200989fd050b76017d8"
server_key_iv: "5ecd98ea062f8a2860a77216049546c4"
server_key_version: "encryptionkey@0"
I20260928 14:53:37.303858 26303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "231e92c575c64d6f8300a88f2af39828"
format_stamp: "Formatted at 2026-09-28 14:53:37 on dist-test-slave-pxl6"
server_key: "e5873ac6cdd74200989fd050b76017d8"
server_key_iv: "5ecd98ea062f8a2860a77216049546c4"
server_key_version: "encryptionkey@0"
I20260928 14:53:37.305325 26303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:53:37.306248 26317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:37.306469 26303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:53:37.306598 26303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "231e92c575c64d6f8300a88f2af39828"
format_stamp: "Formatted at 2026-09-28 14:53:37 on dist-test-slave-pxl6"
server_key: "e5873ac6cdd74200989fd050b76017d8"
server_key_iv: "5ecd98ea062f8a2860a77216049546c4"
server_key_version: "encryptionkey@0"
I20260928 14:53:37.306699 26303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:37.315356 26303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:37.315659 26303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:37.315768 26303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:37.319660 26303 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:36061
I20260928 14:53:37.319751 26370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:36061 every 8 connection(s)
I20260928 14:53:37.320084 26303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:53:37.320812 26371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:37.323706 26371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828: Bootstrap starting.
I20260928 14:53:37.324299 26371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:37.324589 26371 log.cc:824] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:37.325305 26371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828: No bootstrap required, opened a new log
I20260928 14:53:37.326990 26371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231e92c575c64d6f8300a88f2af39828" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36061 } }
I20260928 14:53:37.327167 26371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:37.327214 26371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231e92c575c64d6f8300a88f2af39828, State: Initialized, Role: FOLLOWER
I20260928 14:53:37.327373 26371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231e92c575c64d6f8300a88f2af39828" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36061 } }
I20260928 14:53:37.327459 26371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:53:37.327503 26371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:53:37.327554 26371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:37.328118 26371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231e92c575c64d6f8300a88f2af39828" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36061 } }
I20260928 14:53:37.328241 26371 leader_election.cc:302] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 231e92c575c64d6f8300a88f2af39828; no voters:
I20260928 14:53:37.328423 26371 leader_election.cc:288] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:53:37.328472 26374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:37.328687 26371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:37.328713 26374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828 [term 1 LEADER]: Becoming Leader. State: Replica: 231e92c575c64d6f8300a88f2af39828, State: Running, Role: LEADER
I20260928 14:53:37.328825 26374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231e92c575c64d6f8300a88f2af39828" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36061 } }
I20260928 14:53:37.329255 26375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "231e92c575c64d6f8300a88f2af39828" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231e92c575c64d6f8300a88f2af39828" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36061 } } }
I20260928 14:53:37.329336 26375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828 [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:37.329485 26376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 231e92c575c64d6f8300a88f2af39828. Latest consensus state: current_term: 1 leader_uuid: "231e92c575c64d6f8300a88f2af39828" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231e92c575c64d6f8300a88f2af39828" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36061 } } }
I20260928 14:53:37.329578 26376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828 [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:37.329588 26383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:53:37.330129 26383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:53:37.330319 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 26303
I20260928 14:53:37.330406 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:53:37.330628 20946 external_mini_cluster.cc:1428] Setting key cfad10ece7fd682ab2b5fa7a9d4a3df2
I20260928 14:53:37.332072 26383 catalog_manager.cc:1347] Generated new cluster ID: 9f2403159cbd48748a8e61be36fad6a7
I20260928 14:53:37.332129 26383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:53:37.336707 26393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:37.331095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37916 (local address 127.20.116.190:36061)
0928 14:53:37.332095 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:37.332104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:37.332138 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:53:37.332261 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:37.332268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:37.332323 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:37.332467 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:37.332672 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:37.333576 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:37.334361 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:37.334889 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:37.335050 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:37.335959 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:37.335963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:37.335963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:53:37.336152 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:37.336189 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:53:37.336200 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:37.336280 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:37.336356 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 14:53:37.336544 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":925,"thread_start_us":83,"threads_started":1}
I20260928 14:53:37.337723 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:53:37.350188 26383 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:53:37.350952 26383 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:53:37.359259 26383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 231e92c575c64d6f8300a88f2af39828: Generated new TSK 0
I20260928 14:53:37.359483 26383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:53:37.372253 26393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:37.368158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37918 (local address 127.20.116.190:36061)
0928 14:53:37.368220 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:37.368222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:37.368386 (+ 164us) server_negotiation.cc:407] Connection header received
0928 14:53:37.368424 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:37.368425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:37.368452 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:37.368515 (+ 63us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:53:37.368712 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:37.369241 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:37.370394 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:37.370692 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:37.370749 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:37.370963 (+ 214us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:53:37.372010 (+ 1047us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37091/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 14:53:37.372174 (+ 164us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37918: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37091/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 14:53:37.372331 26393 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37918: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37091/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 14:53:37.372357 26400 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:37.368061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:36061 (local address 127.0.0.1:37918)
0928 14:53:37.368241 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:37.368253 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:37.368328 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:37.368546 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:37.368547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:37.368550 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:53:37.368609 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:37.368612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:37.369331 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:37.369332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:37.370299 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:37.370304 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:37.370776 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:37.370778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:37.370846 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:37.370853 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:53:37.372119 (+ 1266us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37091/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 14:53:37.372241 (+ 122us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.116.190:36061: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37091/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":103,"threads_started":1}
I20260928 14:53:37.374341 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 26303
E20260928 14:53:37.384956 26401 webserver.cc:533] Webserver: sslize failed:
E20260928 14:53:37.385037 26401 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T14:53:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (216 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T14:53:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:53:37Z Disabled control of system clock
I20260928 14:53:37.421285 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:38295
--webserver_interface=127.20.116.190
--webserver_port=0
--builtin_ntp_servers=127.20.116.148:37435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:38295
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607192794321-20946-0/testchainca.cert with env {}
W20260928 14:53:37.505858 26405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:37.506145 26405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:37.506192 26405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:37.507544 26405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:53:37.507637 26405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:37.507658 26405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:37.507678 26405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:53:37.507709 26405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:53:37.507735 26405 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:53:37.507763 26405 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 14:53:37.509377 26405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:37435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:38295
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:38295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607192794321-20946-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:37.509675 26405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:37.509900 26405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:37.515326 26410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:37.515354 26411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:37.515326 26413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:37.515574 26405 server_base.cc:1034] running on GCE node
I20260928 14:53:37.515921 26405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:37.516254 26405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:37.517405 26405 hybrid_clock.cc:648] HybridClock initialized: now 1790607217517385 us; error 40 us; skew 500 ppm
I20260928 14:53:37.518909 26405 webserver.cc:460] Webserver started at http://127.20.116.190:36647/ using document root <none> and password file <none>
I20260928 14:53:37.519179 26405 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:37.519227 26405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:37.519394 26405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:37.520493 26405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "f5103b34b44140979da467a0cb14d10f"
format_stamp: "Formatted at 2026-09-28 14:53:37 on dist-test-slave-pxl6"
server_key: "f013df07fd017e1ada8199e82ead65ed"
server_key_iv: "31ab575c81e8fed9edc55dd6b5d8e702"
server_key_version: "encryptionkey@0"
I20260928 14:53:37.520855 26405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "f5103b34b44140979da467a0cb14d10f"
format_stamp: "Formatted at 2026-09-28 14:53:37 on dist-test-slave-pxl6"
server_key: "f013df07fd017e1ada8199e82ead65ed"
server_key_iv: "31ab575c81e8fed9edc55dd6b5d8e702"
server_key_version: "encryptionkey@0"
I20260928 14:53:37.522164 26405 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:53:37.523072 26419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:37.523288 26405 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:53:37.523384 26405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "f5103b34b44140979da467a0cb14d10f"
format_stamp: "Formatted at 2026-09-28 14:53:37 on dist-test-slave-pxl6"
server_key: "f013df07fd017e1ada8199e82ead65ed"
server_key_iv: "31ab575c81e8fed9edc55dd6b5d8e702"
server_key_version: "encryptionkey@0"
I20260928 14:53:37.523460 26405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:37.544423 26405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:37.544800 26405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:37.545014 26405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:37.549429 26405 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:38295
I20260928 14:53:37.549466 26471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:38295 every 8 connection(s)
I20260928 14:53:37.549860 26405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:53:37.550949 26472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:37.553252 26472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f: Bootstrap starting.
I20260928 14:53:37.553789 26472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:37.554039 26472 log.cc:824] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:37.554756 26472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f: No bootstrap required, opened a new log
I20260928 14:53:37.556387 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 26405
I20260928 14:53:37.556523 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:53:37.556459 26472 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5103b34b44140979da467a0cb14d10f" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 38295 } }
I20260928 14:53:37.556661 26472 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:37.556699 26472 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5103b34b44140979da467a0cb14d10f, State: Initialized, Role: FOLLOWER
I20260928 14:53:37.556716 20946 external_mini_cluster.cc:1428] Setting key da39f52dd72b5430f0abb3c204874fc7
I20260928 14:53:37.556861 26472 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5103b34b44140979da467a0cb14d10f" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 38295 } }
I20260928 14:53:37.556941 26472 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:53:37.556995 26472 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:53:37.557054 26472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:37.557910 26472 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5103b34b44140979da467a0cb14d10f" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 38295 } }
I20260928 14:53:37.558038 26472 leader_election.cc:302] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5103b34b44140979da467a0cb14d10f; no voters:
I20260928 14:53:37.558236 26472 leader_election.cc:288] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:53:37.558332 26477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:37.558491 26472 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:37.558686 26477 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f [term 1 LEADER]: Becoming Leader. State: Replica: f5103b34b44140979da467a0cb14d10f, State: Running, Role: LEADER
I20260928 14:53:37.558816 26477 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5103b34b44140979da467a0cb14d10f" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 38295 } }
I20260928 14:53:37.559263 26478 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5103b34b44140979da467a0cb14d10f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5103b34b44140979da467a0cb14d10f" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 38295 } } }
I20260928 14:53:37.559275 26479 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5103b34b44140979da467a0cb14d10f. Latest consensus state: current_term: 1 leader_uuid: "f5103b34b44140979da467a0cb14d10f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5103b34b44140979da467a0cb14d10f" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 38295 } } }
I20260928 14:53:37.559371 26478 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:37.559388 26479 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:37.559620 26486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:53:37.560163 26486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:53:37.563262 26486 catalog_manager.cc:1347] Generated new cluster ID: 03e8d83c8e944a798e54afa0f3312db2
I20260928 14:53:37.563325 26486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:53:37.563642 26476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:37.557189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.20.116.190:38295)
0928 14:53:37.557441 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:37.557451 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:37.557478 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:53:37.557629 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:37.557634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:37.557699 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:37.557820 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:37.558095 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:37.559936 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:37.561156 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:37.561757 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:37.561885 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:37.562906 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:37.562910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:37.562911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:53:37.563147 (+ 236us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:37.563193 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:53:37.563208 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:37.563298 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:37.563375 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 14:53:37.563457 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":114,"threads_started":1}
I20260928 14:53:37.564606 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:53:37.572378 26486 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:53:37.572926 26486 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:53:37.579981 26486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5103b34b44140979da467a0cb14d10f: Generated new TSK 0
I20260928 14:53:37.580220 26486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:53:38.062232 26476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:37.590159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44114 (local address 127.20.116.190:38295)
0928 14:53:37.590215 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:37.590217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:37.590306 (+ 89us) server_negotiation.cc:407] Connection header received
0928 14:53:37.590351 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:37.590352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:37.590366 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:37.590464 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:53:37.590656 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:37.591464 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:37.592305 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:37.592624 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:37.592701 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:37.592910 (+ 209us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:53:38.061988 (+469078us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 14:53:38.062168 (+ 180us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44114: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":30}
W20260928 14:53:38.062301 26476 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44114: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 14:53:38.062425 26501 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:37.590067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:38295 (local address 127.0.0.1:44114)
0928 14:53:37.590222 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:37.590233 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:37.590297 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:37.590466 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:37.590467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:37.590469 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:53:37.590522 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:37.590524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:37.591576 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:37.591577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:37.592212 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:37.592214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:37.592737 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:37.592738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:37.592786 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:37.592791 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:53:38.062143 (+469352us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 14:53:38.062275 (+ 132us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.116.190:38295: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":102,"threads_started":1}
I20260928 14:53:38.062894 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 26405
2026-09-28T14:53:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (668 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (92 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:53:38 dist-test-slave-pxl6 krb5kdc[26513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:53:38 dist-test-slave-pxl6 krb5kdc[26513](info): set up 2 sockets
Sep 28 14:53:38 dist-test-slave-pxl6 krb5kdc[26513](info): commencing operation
krb5kdc: starting...
W20260928 14:53:39.283929 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:53:39 dist-test-slave-pxl6 krb5kdc[26513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:53:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:53:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:53:39.410847 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:41435
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:41435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:39.494741 26526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:39.495004 26526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:39.495046 26526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:39.496367 26526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:53:39.496444 26526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:39.496470 26526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:39.496518 26526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:53:39.496548 26526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:53:39.498139 26526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:41435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:41435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:39.498448 26526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:39.498719 26526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:39.503731 26534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:39.503724 26532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:39.503731 26531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:39.503930 26526 server_base.cc:1034] running on GCE node
I20260928 14:53:39.504196 26526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:39.504519 26526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:39.505668 26526 hybrid_clock.cc:648] HybridClock initialized: now 1790607219505648 us; error 42 us; skew 500 ppm
Sep 28 14:53:39 dist-test-slave-pxl6 krb5kdc[26513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607219, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:39.512843 26526 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:53:39.513372 26526 webserver.cc:460] Webserver started at http://127.20.116.190:36477/ using document root <none> and password file <none>
I20260928 14:53:39.513630 26526 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:39.513675 26526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:39.513829 26526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:39.515048 26526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "850a5652fdf54ea087320aec2c92f8a7"
format_stamp: "Formatted at 2026-09-28 14:53:39 on dist-test-slave-pxl6"
server_key: "a826a75dc08d421b8affce78b163beb4"
server_key_iv: "38f7b6cee6d981591f24de779f11fae2"
server_key_version: "encryptionkey@0"
I20260928 14:53:39.515430 26526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "850a5652fdf54ea087320aec2c92f8a7"
format_stamp: "Formatted at 2026-09-28 14:53:39 on dist-test-slave-pxl6"
server_key: "a826a75dc08d421b8affce78b163beb4"
server_key_iv: "38f7b6cee6d981591f24de779f11fae2"
server_key_version: "encryptionkey@0"
I20260928 14:53:39.516848 26526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:53:39.517699 26541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:39.517916 26526 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:39.518034 26526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "850a5652fdf54ea087320aec2c92f8a7"
format_stamp: "Formatted at 2026-09-28 14:53:39 on dist-test-slave-pxl6"
server_key: "a826a75dc08d421b8affce78b163beb4"
server_key_iv: "38f7b6cee6d981591f24de779f11fae2"
server_key_version: "encryptionkey@0"
I20260928 14:53:39.518121 26526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:39.536120 26526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:39.538681 26526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:39.538858 26526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:39.543051 26526 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:41435
I20260928 14:53:39.543087 26593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:41435 every 8 connection(s)
I20260928 14:53:39.543464 26526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:53:39.544374 26594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:39.546164 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 26526
I20260928 14:53:39.546293 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:53:39.546471 20946 external_mini_cluster.cc:1428] Setting key 820c8d77eaa76831a0d5e4529b49949e
I20260928 14:53:39.546939 26594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7: Bootstrap starting.
I20260928 14:53:39.547681 26594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:39.548017 26594 log.cc:824] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:39.548797 26594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7: No bootstrap required, opened a new log
I20260928 14:53:39.550696 26594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850a5652fdf54ea087320aec2c92f8a7" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 41435 } }
I20260928 14:53:39.550866 26594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:39.550910 26594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 850a5652fdf54ea087320aec2c92f8a7, State: Initialized, Role: FOLLOWER
I20260928 14:53:39.551071 26594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850a5652fdf54ea087320aec2c92f8a7" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 41435 } }
I20260928 14:53:39.551164 26594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:53:39.551209 26594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:53:39.551260 26594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:39.551823 26594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850a5652fdf54ea087320aec2c92f8a7" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 41435 } }
I20260928 14:53:39.551946 26594 leader_election.cc:302] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 850a5652fdf54ea087320aec2c92f8a7; no voters:
I20260928 14:53:39.552124 26594 leader_election.cc:288] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:53:39.552160 26599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:39.552377 26599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7 [term 1 LEADER]: Becoming Leader. State: Replica: 850a5652fdf54ea087320aec2c92f8a7, State: Running, Role: LEADER
I20260928 14:53:39.552418 26594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:39.553133 26599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850a5652fdf54ea087320aec2c92f8a7" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 41435 } }
Sep 28 14:53:39 dist-test-slave-pxl6 krb5kdc[26513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:39.553762 26600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 850a5652fdf54ea087320aec2c92f8a7. Latest consensus state: current_term: 1 leader_uuid: "850a5652fdf54ea087320aec2c92f8a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850a5652fdf54ea087320aec2c92f8a7" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 41435 } } }
I20260928 14:53:39.553882 26601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "850a5652fdf54ea087320aec2c92f8a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850a5652fdf54ea087320aec2c92f8a7" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 41435 } } }
I20260928 14:53:39.554077 26600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7 [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:39.554087 26601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7 [sys.catalog]: This master's current role is: LEADER
W20260928 14:53:39.554225 26613 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:53:39.554271 26613 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:53:39.554352 26615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:53:39.554625 26615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:53:39.557549 26615 catalog_manager.cc:1347] Generated new cluster ID: 3a5af6b9d0224964a197f48a86650f68
I20260928 14:53:39.557598 26615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:53:39.558030 26597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:39.547105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56586 (local address 127.20.116.190:41435)
0928 14:53:39.547319 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:39.547331 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:39.547369 (+ 38us) server_negotiation.cc:407] Connection header received
0928 14:53:39.548033 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:39.548043 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:39.548110 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:39.548282 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:39.548992 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:39.549886 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:39.550734 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:39.551339 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:39.551447 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:39.554517 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:39.554521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:39.554521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:39.554558 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:39.556350 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:39.556780 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:39.556783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:39.556784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:39.556881 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:39.557082 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:39.557083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:39.557083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:39.557343 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:39.557486 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:39.557647 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 14:53:39.557802 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":87,"threads_started":1}
I20260928 14:53:39.567233 26615 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:53:39.567778 26615 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:53:39.569726 26615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 850a5652fdf54ea087320aec2c92f8a7: Generated new TSK 0
I20260928 14:53:39.569916 26615 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
I20260928 14:53:39.601624 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:41435
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:39.687942 26621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:39.688151 26621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:39.688175 26621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:39.689282 26621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:39.689316 26621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:39.689342 26621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:53:39.690850 26621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:41435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:39.691057 26621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:39.691254 26621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:39.696552 26629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:39.696552 26626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:39.696571 26621 server_base.cc:1034] running on GCE node
W20260928 14:53:39.696569 26627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:39.697038 26621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:39.697333 26621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:39.698504 26621 hybrid_clock.cc:648] HybridClock initialized: now 1790607219698493 us; error 30 us; skew 500 ppm
Sep 28 14:53:39 dist-test-slave-pxl6 krb5kdc[26513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607219, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:39.701156 26621 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:53:39.701668 26621 webserver.cc:460] Webserver started at http://127.20.116.129:40619/ using document root <none> and password file <none>
I20260928 14:53:39.701887 26621 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:39.701928 26621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:39.702018 26621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:39.703107 26621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "aadb8b840b8f43c4b936d97fcadd0746"
format_stamp: "Formatted at 2026-09-28 14:53:39 on dist-test-slave-pxl6"
server_key: "1f8c5ae73f7648e82a2a9aa454e23e12"
server_key_iv: "ebaf7c6e5e0c1591d7f3ef2a876bf55d"
server_key_version: "encryptionkey@0"
I20260928 14:53:39.703414 26621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "aadb8b840b8f43c4b936d97fcadd0746"
format_stamp: "Formatted at 2026-09-28 14:53:39 on dist-test-slave-pxl6"
server_key: "1f8c5ae73f7648e82a2a9aa454e23e12"
server_key_iv: "ebaf7c6e5e0c1591d7f3ef2a876bf55d"
server_key_version: "encryptionkey@0"
I20260928 14:53:39.704710 26621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:53:39.705418 26636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:39.705642 26621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:39.705735 26621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "aadb8b840b8f43c4b936d97fcadd0746"
format_stamp: "Formatted at 2026-09-28 14:53:39 on dist-test-slave-pxl6"
server_key: "1f8c5ae73f7648e82a2a9aa454e23e12"
server_key_iv: "ebaf7c6e5e0c1591d7f3ef2a876bf55d"
server_key_version: "encryptionkey@0"
I20260928 14:53:39.705811 26621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:39.720350 26621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:39.722702 26621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:39.722862 26621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:39.723093 26621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:39.723412 26621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:39.723466 26621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:39.723510 26621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:39.723538 26621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:39.729352 26621 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:42801
I20260928 14:53:39.729403 26749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:42801 every 8 connection(s)
I20260928 14:53:39.729710 26621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:53:39 dist-test-slave-pxl6 krb5kdc[26513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607219, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:39.736922 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 26621
I20260928 14:53:39.737007 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:53:39.737193 20946 external_mini_cluster.cc:1428] Setting key 35a670cd155c62c20000b08e7ec81438
I20260928 14:53:39.739017 26597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:39.730425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:46975 (local address 127.20.116.190:41435)
0928 14:53:39.730509 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:39.730512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:39.730719 (+ 207us) server_negotiation.cc:407] Connection header received
0928 14:53:39.731633 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:39.731641 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:39.731653 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:39.731755 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:39.732339 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:39.732851 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:39.733793 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:39.733985 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:39.734034 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:39.736017 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:39.736018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:39.736018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:39.736025 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:39.737505 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:39.738007 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:39.738008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:39.738008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:39.738060 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:39.738346 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:39.738347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:39.738347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:39.738446 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:39.738496 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:39.738880 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 14:53:39.738955 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:53:39.739300 26752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:39.730375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:41435 (local address 127.20.116.129:46975)
0928 14:53:39.730573 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:39.730626 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:39.731522 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:39.731811 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:39.731814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:39.732055 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:39.732258 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:39.732262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:39.732930 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:39.732930 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:39.733720 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:39.733722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:39.734056 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:39.734057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:39.734156 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:39.734639 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:39.734647 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:39.735935 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:39.737597 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:39.737605 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:39.737606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:39.737912 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:39.738127 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:39.738128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:39.738128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:39.738280 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:39.738510 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:39.738517 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:39.738715 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 14:53:39.738834 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 14:53:39.739012 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":99,"threads_started":1}
I20260928 14:53:39.739892 26750 heartbeater.cc:347] Connected to a master server at 127.20.116.190:41435
I20260928 14:53:39.740007 26750 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:39.740211 26750 heartbeater.cc:511] Master 127.20.116.190:41435 requested a full tablet report, sending...
I20260928 14:53:39.740746 26558 ts_manager.cc:194] Registered new tserver with Master: aadb8b840b8f43c4b936d97fcadd0746 (127.20.116.129:42801)
I20260928 14:53:39.741474 26558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:46975
WARNING: no policy specified for kudu/127.20.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
I20260928 14:53:39.765718 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:41435
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:39.847846 26755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:39.848066 26755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:39.848090 26755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:39.849261 26755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:39.849303 26755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:39.849329 26755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:53:39.850898 26755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:41435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:39.851140 26755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:39.851374 26755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:39.856580 26760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:39.856532 26763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:39.856608 26755 server_base.cc:1034] running on GCE node
W20260928 14:53:39.856671 26761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:39.857024 26755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:39.857280 26755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:39.858448 26755 hybrid_clock.cc:648] HybridClock initialized: now 1790607219858442 us; error 38 us; skew 500 ppm
Sep 28 14:53:39 dist-test-slave-pxl6 krb5kdc[26513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607219, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:39.861151 26755 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:53:39.861624 26755 webserver.cc:460] Webserver started at http://127.20.116.130:43975/ using document root <none> and password file <none>
I20260928 14:53:39.861861 26755 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:39.861933 26755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:39.862092 26755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:39.863234 26755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "16bbee2b4b0043ed99c96cf93cf8e3e3"
format_stamp: "Formatted at 2026-09-28 14:53:39 on dist-test-slave-pxl6"
server_key: "458756acde6914c9332183ffc9ac1ab3"
server_key_iv: "4891dd9b4995b13dd9036d6d029e3436"
server_key_version: "encryptionkey@0"
I20260928 14:53:39.863610 26755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "16bbee2b4b0043ed99c96cf93cf8e3e3"
format_stamp: "Formatted at 2026-09-28 14:53:39 on dist-test-slave-pxl6"
server_key: "458756acde6914c9332183ffc9ac1ab3"
server_key_iv: "4891dd9b4995b13dd9036d6d029e3436"
server_key_version: "encryptionkey@0"
I20260928 14:53:39.864959 26755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:53:39.865742 26770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:39.865937 26755 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:39.866065 26755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "16bbee2b4b0043ed99c96cf93cf8e3e3"
format_stamp: "Formatted at 2026-09-28 14:53:39 on dist-test-slave-pxl6"
server_key: "458756acde6914c9332183ffc9ac1ab3"
server_key_iv: "4891dd9b4995b13dd9036d6d029e3436"
server_key_version: "encryptionkey@0"
I20260928 14:53:39.866144 26755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:39.875571 26755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:39.877931 26755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:39.878099 26755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:39.878338 26755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:39.878722 26755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:39.878778 26755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:39.878822 26755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:39.878849 26755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:39.884793 26755 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:39433
I20260928 14:53:39.884833 26884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:39433 every 8 connection(s)
I20260928 14:53:39.885155 26755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:53:39.890735 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 26755
I20260928 14:53:39.890846 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:53:39.891038 20946 external_mini_cluster.cc:1428] Setting key 6fad7c86f4433ee3190ba9d5e3863099
Sep 28 14:53:39 dist-test-slave-pxl6 krb5kdc[26513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607219, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:39.895606 26887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:39.886009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:41435 (local address 127.20.116.130:34097)
0928 14:53:39.886166 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:39.886201 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:39.887166 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:39.887393 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:39.887397 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:39.887631 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:39.887823 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:39.887827 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:39.888477 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:39.888478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:39.889351 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:39.889354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:39.889756 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:39.889757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:39.889860 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:39.890432 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:39.890441 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:39.892282 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:39.893983 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:39.893992 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:39.893994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:39.894245 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:39.894484 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:39.894485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:39.894486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:39.894703 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:39.895047 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:39.895055 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:39.895183 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 14:53:39.895276 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 14:53:39.895410 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260928 14:53:39.895642 26597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:39.886025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:34097 (local address 127.20.116.190:41435)
0928 14:53:39.886068 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:39.886070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:39.886292 (+ 222us) server_negotiation.cc:407] Connection header received
0928 14:53:39.887274 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:39.887276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:39.887287 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:39.887342 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:39.887895 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:39.888377 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:39.889450 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:39.889675 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:39.889735 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:39.892389 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:39.892391 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:39.892392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:39.892400 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:39.893877 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:39.894328 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:39.894329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:39.894330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:39.894393 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:39.894775 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:39.894776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:39.894777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:39.894937 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:39.895013 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:39.895340 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 14:53:39.895474 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:53:39.896319 26885 heartbeater.cc:347] Connected to a master server at 127.20.116.190:41435
I20260928 14:53:39.896416 26885 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:39.896670 26885 heartbeater.cc:511] Master 127.20.116.190:41435 requested a full tablet report, sending...
I20260928 14:53:39.897111 26558 ts_manager.cc:194] Registered new tserver with Master: 16bbee2b4b0043ed99c96cf93cf8e3e3 (127.20.116.130:39433)
I20260928 14:53:39.897508 26558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:34097
WARNING: no policy specified for kudu/127.20.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
I20260928 14:53:39.918084 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:41435
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:40.001391 26890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:40.001672 26890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:40.001714 26890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:40.002971 26890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:40.003042 26890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:40.003087 26890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:53:40.004624 26890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:41435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:40.004936 26890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:40.005183 26890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:40.010350 26895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:40.010460 26890 server_base.cc:1034] running on GCE node
W20260928 14:53:40.010334 26898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:40.010326 26896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:40.010782 26890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:40.011076 26890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:40.012265 26890 hybrid_clock.cc:648] HybridClock initialized: now 1790607220012249 us; error 25 us; skew 500 ppm
Sep 28 14:53:40 dist-test-slave-pxl6 krb5kdc[26513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607220, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:40.015004 26890 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:53:40.015411 26890 webserver.cc:460] Webserver started at http://127.20.116.131:42955/ using document root <none> and password file <none>
I20260928 14:53:40.015652 26890 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:40.015704 26890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:40.015848 26890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:40.017005 26890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "e9809c2dab394b678602f455aa861492"
format_stamp: "Formatted at 2026-09-28 14:53:40 on dist-test-slave-pxl6"
server_key: "f0122e1350d233527505d9fd3ffb191f"
server_key_iv: "cd4f92a625f0d0ffb146fc017f53fa8d"
server_key_version: "encryptionkey@0"
I20260928 14:53:40.017382 26890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "e9809c2dab394b678602f455aa861492"
format_stamp: "Formatted at 2026-09-28 14:53:40 on dist-test-slave-pxl6"
server_key: "f0122e1350d233527505d9fd3ffb191f"
server_key_iv: "cd4f92a625f0d0ffb146fc017f53fa8d"
server_key_version: "encryptionkey@0"
I20260928 14:53:40.018929 26890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:53:40.019775 26905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:40.019989 26890 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:53:40.020095 26890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "e9809c2dab394b678602f455aa861492"
format_stamp: "Formatted at 2026-09-28 14:53:40 on dist-test-slave-pxl6"
server_key: "f0122e1350d233527505d9fd3ffb191f"
server_key_iv: "cd4f92a625f0d0ffb146fc017f53fa8d"
server_key_version: "encryptionkey@0"
I20260928 14:53:40.020177 26890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:40.041229 26890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:40.043687 26890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:40.043848 26890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:40.044112 26890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:40.044452 26890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:40.044510 26890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:40.044553 26890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:40.044606 26890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:40.050853 26890 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:40629
I20260928 14:53:40.050886 27020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:40629 every 8 connection(s)
I20260928 14:53:40.051258 26890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:53:40.053613 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 26890
I20260928 14:53:40.053740 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:53:40.053934 20946 external_mini_cluster.cc:1428] Setting key da3804397af819785f2ff3d715d13335
Sep 28 14:53:40 dist-test-slave-pxl6 krb5kdc[26513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607220, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:40.062438 26597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:40.052054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:49507 (local address 127.20.116.190:41435)
0928 14:53:40.052145 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:40.052148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:40.052387 (+ 239us) server_negotiation.cc:407] Connection header received
0928 14:53:40.053491 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:40.053492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:40.053503 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:40.053628 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:40.054619 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:40.055323 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:40.056347 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:40.056609 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:40.056674 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:40.059253 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:40.059255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:40.059255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:40.059262 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:40.060765 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:40.061208 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:40.061209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:40.061209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:40.061272 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:40.061800 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:40.061801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:40.061802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:40.061930 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:40.061993 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:40.062287 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 14:53:40.062368 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:53:40.062659 27023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:40.052005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:41435 (local address 127.20.116.131:49507)
0928 14:53:40.052223 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:40.052286 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:40.053361 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:40.053736 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:40.053740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:40.054135 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:40.054522 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:40.054548 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:40.055405 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:40.055407 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:40.056249 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:40.056251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:40.056751 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:40.056752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:40.056869 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:40.057369 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:40.057378 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:40.059160 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:40.060859 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:40.060866 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:40.060868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:40.061127 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:40.061358 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:40.061359 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:40.061359 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:40.061477 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:40.062012 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:40.062019 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:40.062162 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 14:53:40.062254 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 14:53:40.062392 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":141,"threads_started":1}
I20260928 14:53:40.063169 27021 heartbeater.cc:347] Connected to a master server at 127.20.116.190:41435
I20260928 14:53:40.063271 27021 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:40.063424 27021 heartbeater.cc:511] Master 127.20.116.190:41435 requested a full tablet report, sending...
I20260928 14:53:40.063835 26558 ts_manager.cc:194] Registered new tserver with Master: e9809c2dab394b678602f455aa861492 (127.20.116.131:40629)
I20260928 14:53:40.064220 26558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:49507
I20260928 14:53:40.066007 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/test-admin.keytab.
Sep 28 14:53:40 dist-test-slave-pxl6 krb5kdc[26513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:40.084227 20946 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 14:53:40.085974 27028 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:40.085142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42846 (local address 127.20.116.129:42801)
0928 14:53:40.085316 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:40.085321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:40.085332 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:53:40.085361 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:40.085363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:40.085377 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:40.085454 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:40.085854 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42846: BlockingRecv error: recv got EOF from 127.0.0.1:42846 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
W20260928 14:53:40.086059 20946 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/krb5cc' of type FILE
Sep 28 14:53:40 dist-test-slave-pxl6 krb5kdc[26513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:40.087137 20946 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 14:53:40 dist-test-slave-pxl6 krb5kdc[26513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:40.087998 20946 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 14:53:40 dist-test-slave-pxl6 krb5kdc[26513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.129@KRBTEST.COM
I20260928 14:53:40.096525 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:40.088638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42856 (local address 127.20.116.129:42801)
0928 14:53:40.088696 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:40.088698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:40.088795 (+ 97us) server_negotiation.cc:407] Connection header received
0928 14:53:40.088877 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:40.088880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:40.088896 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:40.088958 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:40.089370 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:40.090106 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:40.090956 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:40.091307 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:40.091392 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:40.093615 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:40.093626 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:40.093626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:40.093637 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:40.095318 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:40.095669 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:40.095675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:40.095675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:40.095735 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:40.095936 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:40.095937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:40.095937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:40.096087 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:40.096285 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:40.096331 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 14:53:40.096414 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 14:53:40.098335 27028 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:40.097812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42870 (local address 127.20.116.129:42801)
0928 14:53:40.097844 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:40.097846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:40.097908 (+ 62us) server_negotiation.cc:407] Connection header received
0928 14:53:40.097943 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:40.097943 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:40.097953 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:40.097999 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:40.098289 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42870: BlockingRecv error: recv got EOF from 127.0.0.1:42870 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 14:53:40.098651 20946 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/krb5cc)
Sep 28 14:53:40 dist-test-slave-pxl6 krb5kdc[26513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:40.099774 20946 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 14:53:40 dist-test-slave-pxl6 krb5kdc[26513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:40.100895 20946 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 14:53:40 dist-test-slave-pxl6 krb5kdc[26513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.129@KRBTEST.COM
I20260928 14:53:40.108208 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:40.101522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42878 (local address 127.20.116.129:42801)
0928 14:53:40.101665 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:40.101667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:40.101677 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:53:40.101723 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:40.101724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:40.101734 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:40.101809 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:40.102181 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:40.102764 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:40.103427 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:40.103630 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:40.103687 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:40.105827 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:40.105828 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:40.105829 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:40.105836 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:40.107096 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:40.107442 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:40.107443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:40.107443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:40.107500 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:40.107696 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:40.107697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:40.107697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:40.107811 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:40.107912 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:40.108019 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 14:53:40.108098 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260928 14:53:40.109962 27028 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:40.109373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42892 (local address 127.20.116.129:42801)
0928 14:53:40.109428 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:40.109430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:40.109441 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:53:40.109550 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:40.109550 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:40.109567 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:40.109620 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:40.109899 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42892: BlockingRecv error: recv got EOF from 127.0.0.1:42892 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 14:53:40.110010 20946 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607192794321-20946-0/krb5kdc/krb5cc' of type FILE
Sep 28 14:53:40 dist-test-slave-pxl6 krb5kdc[26513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:40.111109 20946 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 14:53:40 dist-test-slave-pxl6 krb5kdc[26513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:40.111945 20946 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 14:53:40 dist-test-slave-pxl6 krb5kdc[26513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.129@KRBTEST.COM
I20260928 14:53:40.118887 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:40.112411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42902 (local address 127.20.116.129:42801)
0928 14:53:40.112467 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:40.112469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:40.112583 (+ 114us) server_negotiation.cc:407] Connection header received
0928 14:53:40.112663 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:40.112663 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:40.112672 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:40.112728 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:40.113087 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:40.113546 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:40.114243 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:40.114438 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:40.114515 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:40.116543 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:40.116545 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:40.116545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:40.116552 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:40.117717 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:40.118058 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:40.118059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:40.118060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:40.118118 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:40.118318 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:40.118319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:40.118319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:40.118419 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:40.118518 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:40.118712 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 14:53:40.118786 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 14:53:40.119524 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 26621
I20260928 14:53:40.125133 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 26755
I20260928 14:53:40.129781 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 26890
I20260928 14:53:40.134020 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 26526
2026-09-28T14:53:40Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1889 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:53:40 dist-test-slave-pxl6 krb5kdc[27041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:53:40 dist-test-slave-pxl6 krb5kdc[27041](info): set up 2 sockets
Sep 28 14:53:40 dist-test-slave-pxl6 krb5kdc[27041](info): commencing operation
krb5kdc: starting...
W20260928 14:53:41.167932 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:53:41 dist-test-slave-pxl6 krb5kdc[27041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:53:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:53:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.190.keytab.
Entry for principal oryx/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.190.keytab.
I20260928 14:53:41.294039 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:35947
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.190.keytab
--principal=oryx/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:35947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:41.378852 27054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:41.379135 27054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:41.379176 27054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:41.380421 27054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:53:41.380501 27054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:41.380524 27054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:41.380545 27054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:53:41.380574 27054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:53:41.382212 27054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:35947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.190.keytab
--principal=oryx/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:35947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:41.382555 27054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:41.382817 27054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:41.388019 27060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:41.388062 27062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:41.388038 27059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:41.388347 27054 server_base.cc:1034] running on GCE node
I20260928 14:53:41.388648 27054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:41.388971 27054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:41.390174 27054 hybrid_clock.cc:648] HybridClock initialized: now 1790607221390143 us; error 56 us; skew 500 ppm
Sep 28 14:53:41 dist-test-slave-pxl6 krb5kdc[27041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607221, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:41.392980 27054 init.cc:377] Logged in from keytab as oryx/127.20.116.190@KRBTEST.COM (short username oryx)
I20260928 14:53:41.393549 27054 webserver.cc:460] Webserver started at http://127.20.116.190:36933/ using document root <none> and password file <none>
I20260928 14:53:41.393792 27054 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:41.393872 27054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:41.394047 27054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:41.395301 27054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "2947c96f35da45d4b4bb59aeda6a1e0d"
format_stamp: "Formatted at 2026-09-28 14:53:41 on dist-test-slave-pxl6"
server_key: "0cb0ac975d2347e6859ab81ce8b19919"
server_key_iv: "13079600cbf4340ee5d0080013d6e028"
server_key_version: "encryptionkey@0"
I20260928 14:53:41.395701 27054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "2947c96f35da45d4b4bb59aeda6a1e0d"
format_stamp: "Formatted at 2026-09-28 14:53:41 on dist-test-slave-pxl6"
server_key: "0cb0ac975d2347e6859ab81ce8b19919"
server_key_iv: "13079600cbf4340ee5d0080013d6e028"
server_key_version: "encryptionkey@0"
I20260928 14:53:41.397248 27054 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 14:53:41.398273 27069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:41.398612 27054 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:41.398749 27054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "2947c96f35da45d4b4bb59aeda6a1e0d"
format_stamp: "Formatted at 2026-09-28 14:53:41 on dist-test-slave-pxl6"
server_key: "0cb0ac975d2347e6859ab81ce8b19919"
server_key_iv: "13079600cbf4340ee5d0080013d6e028"
server_key_version: "encryptionkey@0"
I20260928 14:53:41.398852 27054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:41.415390 27054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:41.417632 27054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:41.417817 27054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:41.422407 27054 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:35947
I20260928 14:53:41.422457 27121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:35947 every 8 connection(s)
I20260928 14:53:41.422880 27054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:53:41.423993 27122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:41.426415 27122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d: Bootstrap starting.
I20260928 14:53:41.427116 27122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:41.427424 27122 log.cc:824] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:41.428340 27122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d: No bootstrap required, opened a new log
I20260928 14:53:41.429536 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 27054
I20260928 14:53:41.429661 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:53:41.429862 20946 external_mini_cluster.cc:1428] Setting key 269a86bd77096dccafb09236c29bb333
I20260928 14:53:41.430498 27122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } }
I20260928 14:53:41.430753 27122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:41.430806 27122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2947c96f35da45d4b4bb59aeda6a1e0d, State: Initialized, Role: FOLLOWER
I20260928 14:53:41.430948 27122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } }
I20260928 14:53:41.431041 27122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:53:41.431083 27122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:53:41.431131 27122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:41.432075 27122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } }
I20260928 14:53:41.432204 27122 leader_election.cc:302] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2947c96f35da45d4b4bb59aeda6a1e0d; no voters:
I20260928 14:53:41.432410 27122 leader_election.cc:288] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:53:41.432442 27127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:41.432559 27127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [term 1 LEADER]: Becoming Leader. State: Replica: 2947c96f35da45d4b4bb59aeda6a1e0d, State: Running, Role: LEADER
I20260928 14:53:41.432732 27122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:41.432706 27127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } }
I20260928 14:53:41.433123 27127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2947c96f35da45d4b4bb59aeda6a1e0d. Latest consensus state: current_term: 1 leader_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } }
I20260928 14:53:41.433424 27127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:41.433311 27128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } }
I20260928 14:53:41.433912 27128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:41.433991 27137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:53:41.434590 27137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 14:53:41 dist-test-slave-pxl6 krb5kdc[27041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM
I20260928 14:53:41.439157 27137 catalog_manager.cc:1347] Generated new cluster ID: b4b97e5e908a4355897c5dbba5724d5a
I20260928 14:53:41.439239 27137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:53:41.442368 27126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:41.430500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32892 (local address 127.20.116.190:35947)
0928 14:53:41.430747 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:41.430758 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:41.430821 (+ 63us) server_negotiation.cc:407] Connection header received
0928 14:53:41.430955 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:41.430962 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:41.431025 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:41.431188 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:41.431939 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:41.433309 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:41.434365 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:41.434991 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:41.435115 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:41.437944 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:41.437949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:41.437949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:41.437964 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:41.440107 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:41.440554 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:41.440557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:41.440558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:41.440674 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:41.440910 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:41.440911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:41.440912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:41.441228 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:41.441851 (+ 623us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:41.441937 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 14:53:41.442104 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":127,"threads_started":1}
I20260928 14:53:41.447512 27137 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:53:41.448232 27137 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:53:41.453447 27137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d: Generated new TSK 0
I20260928 14:53:41.453670 27137 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.129.keytab.
Entry for principal oryx/127.20.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.129.keytab.
I20260928 14:53:41.482815 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35947
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.129.keytab
--principal=oryx/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:41.568450 27149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:41.568681 27149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:41.568707 27149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:41.569955 27149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:41.570014 27149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:41.570042 27149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:53:41.571622 27149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.129.keytab
--principal=oryx/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:41.571911 27149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:41.572180 27149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:41.577684 27154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:41.577626 27157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:41.577687 27155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:41.577730 27149 server_base.cc:1034] running on GCE node
I20260928 14:53:41.578023 27149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:41.578274 27149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:41.579439 27149 hybrid_clock.cc:648] HybridClock initialized: now 1790607221579444 us; error 47 us; skew 500 ppm
Sep 28 14:53:41 dist-test-slave-pxl6 krb5kdc[27041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607221, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:41.582003 27149 init.cc:377] Logged in from keytab as oryx/127.20.116.129@KRBTEST.COM (short username oryx)
I20260928 14:53:41.582453 27149 webserver.cc:460] Webserver started at http://127.20.116.129:44905/ using document root <none> and password file <none>
I20260928 14:53:41.582727 27149 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:41.582798 27149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:41.582947 27149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:41.584061 27149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "dd1d5fa791f449278948c4abbb832e43"
format_stamp: "Formatted at 2026-09-28 14:53:41 on dist-test-slave-pxl6"
server_key: "109c903c742f51a56963d285d7fcd439"
server_key_iv: "a92324a2c5ad304a2a5de50e6910bddf"
server_key_version: "encryptionkey@0"
I20260928 14:53:41.584444 27149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "dd1d5fa791f449278948c4abbb832e43"
format_stamp: "Formatted at 2026-09-28 14:53:41 on dist-test-slave-pxl6"
server_key: "109c903c742f51a56963d285d7fcd439"
server_key_iv: "a92324a2c5ad304a2a5de50e6910bddf"
server_key_version: "encryptionkey@0"
I20260928 14:53:41.585839 27149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:53:41.586714 27164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:41.586930 27149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:41.587033 27149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "dd1d5fa791f449278948c4abbb832e43"
format_stamp: "Formatted at 2026-09-28 14:53:41 on dist-test-slave-pxl6"
server_key: "109c903c742f51a56963d285d7fcd439"
server_key_iv: "a92324a2c5ad304a2a5de50e6910bddf"
server_key_version: "encryptionkey@0"
I20260928 14:53:41.587118 27149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:41.601303 27149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:41.603976 27149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:41.604153 27149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:41.604426 27149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:41.604775 27149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:41.604830 27149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:41.604873 27149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:41.604902 27149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:41.610962 27149 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:35257
I20260928 14:53:41.611011 27278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:35257 every 8 connection(s)
I20260928 14:53:41.611359 27149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:53:41 dist-test-slave-pxl6 krb5kdc[27041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607221, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM
I20260928 14:53:41.618284 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 27149
I20260928 14:53:41.618407 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:53:41.618693 20946 external_mini_cluster.cc:1428] Setting key 3ab6ba165e057b8f4349f8affdd6fe13
I20260928 14:53:41.621889 27126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:41.612231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:45885 (local address 127.20.116.190:35947)
0928 14:53:41.612282 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:41.612284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:41.612537 (+ 253us) server_negotiation.cc:407] Connection header received
0928 14:53:41.613103 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:41.613104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:41.613122 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:41.613177 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:41.614170 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:41.614756 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:41.615818 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:41.616035 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:41.616123 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:41.618580 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:41.618582 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:41.618583 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:41.618593 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:41.620188 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:41.620765 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:41.620766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:41.620766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:41.620819 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:41.621146 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:41.621147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:41.621147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:41.621300 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:41.621372 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:41.621666 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 14:53:41.621767 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:53:41.621980 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:41.612205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.129:45885)
0928 14:53:41.612405 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:41.612446 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:41.612984 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:41.613269 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:41.613272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:41.613847 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:41.614055 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:41.614060 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:41.614872 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:41.614873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:41.615710 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:41.615712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:41.616138 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:41.616140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:41.616279 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:41.616855 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:41.616863 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:41.618431 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:41.620282 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:41.620290 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:41.620293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:41.620664 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:41.620892 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:41.620892 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:41.620893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:41.621066 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:41.621371 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:41.621377 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:41.621512 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 14:53:41.621623 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 14:53:41.621760 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":98,"threads_started":1}
I20260928 14:53:41.622573 27279 heartbeater.cc:347] Connected to a master server at 127.20.116.190:35947
I20260928 14:53:41.622686 27279 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:41.622884 27279 heartbeater.cc:511] Master 127.20.116.190:35947 requested a full tablet report, sending...
I20260928 14:53:41.623559 27086 ts_manager.cc:194] Registered new tserver with Master: dd1d5fa791f449278948c4abbb832e43 (127.20.116.129:35257)
I20260928 14:53:41.624578 27086 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:45885
WARNING: no policy specified for oryx/127.20.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.130.keytab.
Entry for principal oryx/127.20.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.130.keytab.
I20260928 14:53:41.646332 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35947
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.130.keytab
--principal=oryx/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:41.734014 27284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:41.734236 27284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:41.734263 27284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:41.735581 27284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:41.735630 27284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:41.735658 27284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:53:41.737214 27284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.130.keytab
--principal=oryx/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:41.737481 27284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:41.737754 27284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:41.743371 27292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:41.743383 27289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:41.743398 27284 server_base.cc:1034] running on GCE node
W20260928 14:53:41.743371 27290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:41.743901 27284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:41.744169 27284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:41.745370 27284 hybrid_clock.cc:648] HybridClock initialized: now 1790607221745364 us; error 59 us; skew 500 ppm
Sep 28 14:53:41 dist-test-slave-pxl6 krb5kdc[27041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607221, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:41.748121 27284 init.cc:377] Logged in from keytab as oryx/127.20.116.130@KRBTEST.COM (short username oryx)
I20260928 14:53:41.748623 27284 webserver.cc:460] Webserver started at http://127.20.116.130:38111/ using document root <none> and password file <none>
I20260928 14:53:41.748862 27284 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:41.748930 27284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:41.749068 27284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:41.750147 27284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "d18126ddd7c840049afba377de0fa029"
format_stamp: "Formatted at 2026-09-28 14:53:41 on dist-test-slave-pxl6"
server_key: "43448885792c1878c26edf00d206550b"
server_key_iv: "f26878272862b69c11a35d78d3cc68a1"
server_key_version: "encryptionkey@0"
I20260928 14:53:41.750577 27284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "d18126ddd7c840049afba377de0fa029"
format_stamp: "Formatted at 2026-09-28 14:53:41 on dist-test-slave-pxl6"
server_key: "43448885792c1878c26edf00d206550b"
server_key_iv: "f26878272862b69c11a35d78d3cc68a1"
server_key_version: "encryptionkey@0"
I20260928 14:53:41.752036 27284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:41.752880 27299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:41.753130 27284 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:41.753237 27284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "d18126ddd7c840049afba377de0fa029"
format_stamp: "Formatted at 2026-09-28 14:53:41 on dist-test-slave-pxl6"
server_key: "43448885792c1878c26edf00d206550b"
server_key_iv: "f26878272862b69c11a35d78d3cc68a1"
server_key_version: "encryptionkey@0"
I20260928 14:53:41.753332 27284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:41.777318 27284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:41.779639 27284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:41.779843 27284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:41.780109 27284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:41.780457 27284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:41.780516 27284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:41.780583 27284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:41.780613 27284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:41.786934 27284 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:35679
I20260928 14:53:41.786980 27412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:35679 every 8 connection(s)
I20260928 14:53:41.787326 27284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:53:41.791769 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 27284
I20260928 14:53:41.791872 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:53:41.792068 20946 external_mini_cluster.cc:1428] Setting key 696ea2af53063252e844f52af82c7f21
Sep 28 14:53:41 dist-test-slave-pxl6 krb5kdc[27041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607221, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM
I20260928 14:53:41.797834 27126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:41.788113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:53365 (local address 127.20.116.190:35947)
0928 14:53:41.788190 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:41.788192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:41.788407 (+ 215us) server_negotiation.cc:407] Connection header received
0928 14:53:41.788921 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:41.788922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:41.788933 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:41.788990 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:41.789996 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:41.790660 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:41.791721 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:41.792029 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:41.792112 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:41.794451 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:41.794453 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:41.794453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:41.794463 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:41.796063 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:41.796684 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:41.796685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:41.796685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:41.796767 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:41.797084 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:41.797085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:41.797085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:41.797191 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:41.797244 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:41.797683 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 14:53:41.797765 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:53:41.798135 27415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:41.788089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.130:53365)
0928 14:53:41.788281 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:41.788321 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:41.788824 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:41.789062 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:41.789066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:41.789678 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:41.789886 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:41.789891 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:41.790750 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:41.790752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:41.791629 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:41.791631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:41.792136 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:41.792138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:41.792277 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:41.792889 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:41.792898 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:41.794352 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:41.796159 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:41.796167 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:41.796170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:41.796542 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:41.796844 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:41.796845 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:41.796846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:41.797007 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:41.797263 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:41.797270 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:41.797459 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 14:53:41.797609 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 14:53:41.797817 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":110,"threads_started":1}
I20260928 14:53:41.798780 27413 heartbeater.cc:347] Connected to a master server at 127.20.116.190:35947
I20260928 14:53:41.798884 27413 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:41.799113 27413 heartbeater.cc:511] Master 127.20.116.190:35947 requested a full tablet report, sending...
I20260928 14:53:41.799572 27086 ts_manager.cc:194] Registered new tserver with Master: d18126ddd7c840049afba377de0fa029 (127.20.116.130:35679)
I20260928 14:53:41.799968 27086 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:53365
WARNING: no policy specified for oryx/127.20.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.131.keytab.
Entry for principal oryx/127.20.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.131.keytab.
I20260928 14:53:41.821604 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35947
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.131.keytab
--principal=oryx/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:41.908051 27418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:41.908313 27418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:41.908355 27418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:41.909631 27418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:41.909704 27418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:41.909751 27418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:53:41.911279 27418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.131.keytab
--principal=oryx/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:41.911584 27418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:41.911834 27418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:41.917088 27427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:41.917102 27425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:41.917088 27424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:41.917375 27418 server_base.cc:1034] running on GCE node
I20260928 14:53:41.917609 27418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:41.917820 27418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:41.918992 27418 hybrid_clock.cc:648] HybridClock initialized: now 1790607221918982 us; error 37 us; skew 500 ppm
Sep 28 14:53:41 dist-test-slave-pxl6 krb5kdc[27041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607221, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:41.921571 27418 init.cc:377] Logged in from keytab as oryx/127.20.116.131@KRBTEST.COM (short username oryx)
I20260928 14:53:41.922143 27418 webserver.cc:460] Webserver started at http://127.20.116.131:34011/ using document root <none> and password file <none>
I20260928 14:53:41.922434 27418 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:41.922487 27418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:41.922654 27418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:41.923705 27418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "c0148ddeb6e042fbb5ad121e91de2860"
format_stamp: "Formatted at 2026-09-28 14:53:41 on dist-test-slave-pxl6"
server_key: "484d1eea8cb0bb6c4db3fac2a743e393"
server_key_iv: "53f1717e9eb5b2bf9e3fdd33578be44e"
server_key_version: "encryptionkey@0"
I20260928 14:53:41.924041 27418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "c0148ddeb6e042fbb5ad121e91de2860"
format_stamp: "Formatted at 2026-09-28 14:53:41 on dist-test-slave-pxl6"
server_key: "484d1eea8cb0bb6c4db3fac2a743e393"
server_key_iv: "53f1717e9eb5b2bf9e3fdd33578be44e"
server_key_version: "encryptionkey@0"
I20260928 14:53:41.925360 27418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:53:41.926066 27434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:41.926280 27418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:41.926373 27418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "c0148ddeb6e042fbb5ad121e91de2860"
format_stamp: "Formatted at 2026-09-28 14:53:41 on dist-test-slave-pxl6"
server_key: "484d1eea8cb0bb6c4db3fac2a743e393"
server_key_iv: "53f1717e9eb5b2bf9e3fdd33578be44e"
server_key_version: "encryptionkey@0"
I20260928 14:53:41.926429 27418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:41.952734 27418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:41.955235 27418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:41.955354 27418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:41.955542 27418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:41.955907 27418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:41.955966 27418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:41.956010 27418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:41.956035 27418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:41.961966 27418 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:42409
I20260928 14:53:41.962052 27547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:42409 every 8 connection(s)
I20260928 14:53:41.962399 27418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:53:41.967456 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 27418
I20260928 14:53:41.967564 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:53:41.967773 20946 external_mini_cluster.cc:1428] Setting key 626734c0a69a91466799d0e88d69c9b9
Sep 28 14:53:41 dist-test-slave-pxl6 krb5kdc[27041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607221, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM
I20260928 14:53:41.971959 27126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:41.963218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:47997 (local address 127.20.116.190:35947)
0928 14:53:41.963288 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:41.963290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:41.963504 (+ 214us) server_negotiation.cc:407] Connection header received
0928 14:53:41.964151 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:41.964152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:41.964164 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:41.964261 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:41.965238 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:41.965737 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:41.966796 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:41.966981 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:41.967034 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:41.969304 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:41.969305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:41.969305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:41.969312 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:41.970797 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:41.971160 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:41.971160 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:41.971161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:41.971225 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:41.971452 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:41.971452 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:41.971453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:41.971553 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:41.971602 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:41.971835 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 14:53:41.971894 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:53:41.972162 27550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:41.963186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.131:47997)
0928 14:53:41.963373 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:41.963417 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:41.964013 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:41.964341 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:41.964344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:41.964928 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:41.965146 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:41.965151 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:41.965816 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:41.965817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:41.966722 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:41.966725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:41.967061 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:41.967062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:41.967188 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:41.967790 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:41.967799 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:41.969224 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:41.970866 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:41.970872 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:41.970874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:41.971089 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:41.971283 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:41.971284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:41.971284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:41.971398 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:41.971609 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:41.971614 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:41.971727 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 14:53:41.971809 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 14:53:41.971935 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
I20260928 14:53:41.972683 27548 heartbeater.cc:347] Connected to a master server at 127.20.116.190:35947
I20260928 14:53:41.972779 27548 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:41.972929 27548 heartbeater.cc:511] Master 127.20.116.190:35947 requested a full tablet report, sending...
I20260928 14:53:41.973301 27086 ts_manager.cc:194] Registered new tserver with Master: c0148ddeb6e042fbb5ad121e91de2860 (127.20.116.131:42409)
I20260928 14:53:41.973672 27086 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:47997
I20260928 14:53:41.979449 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.116.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.116.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.116.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.189.keytab.
Entry for principal oryx/127.20.116.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.189.keytab.
I20260928 14:53:42.006210 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.189:37691
--webserver_interface=127.20.116.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.189.keytab
--principal=oryx/127.20.116.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:35947,127.20.116.189:37691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:42.090500 27553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:42.090796 27553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:42.090833 27553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:42.092087 27553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:53:42.092154 27553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:42.092178 27553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:42.092199 27553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:53:42.092231 27553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:53:42.093724 27553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:35947,127.20.116.189:37691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.189.keytab
--principal=oryx/127.20.116.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.189:37691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.116.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
W20260928 14:53:42.093868 27553 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.20.116.190:35947,127.20.116.189:37691'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 14:53:42.094031 27553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:42.094260 27553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:42.099586 27559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:42.099570 27558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:42.099702 27553 server_base.cc:1034] running on GCE node
W20260928 14:53:42.099570 27561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:42.100129 27553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:42.100443 27553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:42.101620 27553 hybrid_clock.cc:648] HybridClock initialized: now 1790607222101612 us; error 35 us; skew 500 ppm
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607222, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:42.104271 27553 init.cc:377] Logged in from keytab as oryx/127.20.116.189@KRBTEST.COM (short username oryx)
I20260928 14:53:42.104768 27553 webserver.cc:460] Webserver started at http://127.20.116.189:36041/ using document root <none> and password file <none>
I20260928 14:53:42.105020 27553 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:42.105089 27553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:42.105242 27553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:42.106323 27553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/data/instance:
uuid: "51131b65b43344d59beff206ed38e6bd"
format_stamp: "Formatted at 2026-09-28 14:53:42 on dist-test-slave-pxl6"
server_key: "db18a5bcc76910a06e469a156299535a"
server_key_iv: "a97a74443e19da2e22a9e6ab9785d050"
server_key_version: "encryptionkey@0"
I20260928 14:53:42.106818 27553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/wal/instance:
uuid: "51131b65b43344d59beff206ed38e6bd"
format_stamp: "Formatted at 2026-09-28 14:53:42 on dist-test-slave-pxl6"
server_key: "db18a5bcc76910a06e469a156299535a"
server_key_iv: "a97a74443e19da2e22a9e6ab9785d050"
server_key_version: "encryptionkey@0"
I20260928 14:53:42.108573 27553 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 14:53:42.109472 27568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:42.109663 27553 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:53:42.109771 27553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/wal
uuid: "51131b65b43344d59beff206ed38e6bd"
format_stamp: "Formatted at 2026-09-28 14:53:42 on dist-test-slave-pxl6"
server_key: "db18a5bcc76910a06e469a156299535a"
server_key_iv: "a97a74443e19da2e22a9e6ab9785d050"
server_key_version: "encryptionkey@0"
I20260928 14:53:42.109845 27553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:42.119596 27553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:42.122056 27553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:42.122226 27553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:42.126513 27553 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.189:37691
I20260928 14:53:42.126605 27620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.189:37691 every 8 connection(s)
I20260928 14:53:42.127025 27553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/data/info.pb
I20260928 14:53:42.128093 27621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:42.129830 27621 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:53:42.131690 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 27553
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607222, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.189@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.116.189@KRBTEST.COM
I20260928 14:53:42.140990 27126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.130637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32898 (local address 127.20.116.190:35947)
0928 14:53:42.130709 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.130711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.130895 (+ 184us) server_negotiation.cc:407] Connection header received
0928 14:53:42.131353 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.131354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.131365 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.131419 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.132289 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.132932 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.134170 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.134469 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.134575 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.137148 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.137150 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.137150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.137159 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.138901 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.139520 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.139521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.139521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.139636 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.139964 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.139965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.139965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.140141 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.140345 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.140795 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.140889 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 14:53:42.141234 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.130601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.0.0.1:32898)
0928 14:53:42.130773 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.130809 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.131268 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.131487 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.131490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.131864 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.132141 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.132149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.133040 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.133041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.134076 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.134079 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.134574 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.134575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.134709 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.135255 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.135264 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.137020 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.138998 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.139005 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.139008 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.139410 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.139723 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.139724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.139725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.139863 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.140340 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.140346 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.140488 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.140766 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.140951 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260928 14:53:42.141237 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.132228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33138 (local address 127.20.116.189:37691)
0928 14:53:42.132400 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.132405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.132428 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:53:42.132452 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.132454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.132474 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.132572 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.133198 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.134138 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.135027 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.135325 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.135416 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.138005 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.138014 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.138014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.138021 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.139948 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.140276 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.140279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.140279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.140345 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.140584 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.140585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.140585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.140728 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.140910 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.140964 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.141052 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
I20260928 14:53:42.141968 27621 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607222, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.189@KRBTEST.COM for oryx/127.20.116.189@KRBTEST.COM
I20260928 14:53:42.149484 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.142250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:37691 (local address 127.0.0.1:33154)
0928 14:53:42.142303 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.142315 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.142357 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.142518 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.142518 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.142707 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.142753 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.142755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.143369 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.143370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.143872 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.143874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.144260 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.144261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.144312 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.144724 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.144729 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.145909 (+ 1180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.148406 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.148407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.148407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.148572 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.148850 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.148851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.148851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.148903 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.149252 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.149252 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.149290 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.149337 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.149411 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 14:53:42.149610 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.142363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.20.116.189:37691)
0928 14:53:42.142410 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.142411 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.142419 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:53:42.142436 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.142437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.142452 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.142502 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.142822 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.143298 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.143956 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.144184 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.144252 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.146061 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.146062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.146062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.146081 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.148284 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.148662 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.148663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.148663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.148768 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.148951 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.148952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.148952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.149189 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.149257 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.149406 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.149496 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 14:53:42.151165 27621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd: Bootstrap starting.
I20260928 14:53:42.152029 27621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:42.152320 27621 log.cc:824] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:42.153173 27621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd: No bootstrap required, opened a new log
I20260928 14:53:42.154786 27621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } }
I20260928 14:53:42.154913 27621 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:42.154937 27621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51131b65b43344d59beff206ed38e6bd, State: Initialized, Role: FOLLOWER
I20260928 14:53:42.155069 27621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } }
I20260928 14:53:42.155396 27621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:42.155359 27627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } } }
I20260928 14:53:42.155622 27627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [sys.catalog]: This master's current role is: FOLLOWER
W20260928 14:53:42.157366 27638 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:53:42.157429 27638 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:53:42.159119 27553 master_runner.cc:428] Detected that this master 51131b65b43344d59beff206ed38e6bd is joining an existing cluster
I20260928 14:53:42.159161 27553 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.116.189:37691
I20260928 14:53:42.167150 27126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.160421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32900 (local address 127.20.116.190:35947)
0928 14:53:42.160466 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.160468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.160565 (+ 97us) server_negotiation.cc:407] Connection header received
0928 14:53:42.160591 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.160592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.160604 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.160654 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.161109 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.161815 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.162576 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.162841 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.162900 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.164099 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.164101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.164101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.164108 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.165816 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.166215 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.166216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.166216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.166280 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.166521 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.166523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.166523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.166700 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.166749 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.166964 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.167048 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:53:42.167433 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.160230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33168 (local address 127.20.116.189:37691)
0928 14:53:42.160288 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.160290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.160299 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:53:42.160402 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.160402 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.160412 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.160461 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.160932 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.161670 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.162752 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.163019 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.163083 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.164268 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.164269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.164270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.164276 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.165991 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.166405 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.166406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.166406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.166453 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.166818 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.166819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.166819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.166937 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.167016 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.167147 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.167350 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":132,"server-negotiator.queue_time_us":38}
I20260928 14:53:42.167619 27645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.160353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.0.0.1:32900)
0928 14:53:42.160469 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.160479 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.160534 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.160671 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.160672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.160924 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.161014 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.161017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.161890 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.161891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.162460 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.162461 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.162967 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.162968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.163024 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.163556 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.163561 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.163982 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.165912 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.165913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.165914 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.166135 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.166363 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.166363 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.166364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.166431 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.166800 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.166801 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.166848 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.167256 (+ 408us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.167420 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":53,"thread_start_us":68,"threads_started":1}
I20260928 14:53:42.167680 27644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.160063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:37691 (local address 127.0.0.1:33168)
0928 14:53:42.160217 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.160227 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.160301 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.160509 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.160509 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.160773 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.160857 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.160860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.161758 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.161759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.162670 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.162673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.163099 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.163100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.163169 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.163746 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.163751 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.164171 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.166084 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.166084 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.166085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.166324 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.166643 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.166645 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.166646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.166734 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.167009 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.167009 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.167047 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.167115 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.167513 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607221, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.190@KRBTEST.COM for oryx/127.20.116.189@KRBTEST.COM
I20260928 14:53:42.177762 27646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.170009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:37691 (local address 127.0.0.1:33180)
0928 14:53:42.170186 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.170237 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.170320 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.170465 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.170468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.170833 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.170895 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.170898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.171476 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.171477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.172446 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.172448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.172826 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.172827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.172911 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.173328 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.173333 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.174945 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.176624 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.176632 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.176635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.176842 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.177075 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.177076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.177077 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.177143 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.177447 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.177453 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.177502 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.177585 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.177686 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":122,"threads_started":1}
I20260928 14:53:42.177793 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.170062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.20.116.189:37691)
0928 14:53:42.170118 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.170120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.170311 (+ 191us) server_negotiation.cc:407] Connection header received
0928 14:53:42.170395 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.170395 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.170404 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.170459 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.170971 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.171401 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.172525 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.172752 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.172806 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.175026 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.175027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.175028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.175035 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.176533 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.176928 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.176929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.176929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.176987 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.177232 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.177233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.177233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.177364 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.177434 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.177627 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.177701 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:53:42.178501 27086 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.116.189:37691: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "51131b65b43344d59beff206ed38e6bd"
member_type: NON_VOTER
last_known_addr {
host: "127.20.116.189"
port: 37691
}
attrs {
promote: true
}
}
dest_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d"
cas_config_opid_index: -1
I20260928 14:53:42.178900 27086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true } }
I20260928 14:53:42.179430 27648 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 51131b65b43344d59beff206ed38e6bd (127.20.116.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true } } }
I20260928 14:53:42.179823 27595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:42.180534 27648 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.116.189:37691
I20260928 14:53:42.180620 27648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true } } }
I20260928 14:53:42.180662 27595 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 1 FOLLOWER]: Refusing update from remote peer 2947c96f35da45d4b4bb59aeda6a1e0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 14:53:42.180689 27648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:42.180974 27553 master.cc:537] Master@127.20.116.189:37691 shutting down...
I20260928 14:53:42.181556 27647 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 14:53:42.182286 27072 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 14:53:42.183151 27072 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d -> Peer 51131b65b43344d59beff206ed38e6bd (127.20.116.189:37691): Couldn't send request to peer 51131b65b43344d59beff206ed38e6bd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 14:53:42.183981 27553 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:42.184074 27553 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:42.184125 27553 tablet_replica.cc:330] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd: stopping tablet replica
I20260928 14:53:42.197005 27553 master.cc:552] Master@127.20.116.189:37691 shutdown complete.
I20260928 14:53:42.197067 27553 master_runner.cc:305] Clearing existing system tablet
I20260928 14:53:42.197211 27553 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:42.197259 27553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:42.198010 27553 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 14:53:42.198583 27652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:42.198751 27553 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:42.198817 27553 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:42.198853 27553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/wal
uuid: "51131b65b43344d59beff206ed38e6bd"
format_stamp: "Formatted at 2026-09-28 14:53:42 on dist-test-slave-pxl6"
server_key: "db18a5bcc76910a06e469a156299535a"
server_key_iv: "a97a74443e19da2e22a9e6ab9785d050"
server_key_version: "encryptionkey@0"
I20260928 14:53:42.199479 27553 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:42.200999 27553 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 14:53:42.201067 27553 log.cc:1196] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 14:53:42.201270 27553 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd: Deleting consensus metadata
I20260928 14:53:42.201401 27553 master_runner.cc:315] Copying system tablet from 127.20.116.190:35947
I20260928 14:53:42.202216 27553 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd: tablet copy: Beginning tablet copy session from remote peer at address 127.20.116.190:35947
I20260928 14:53:42.208985 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.202560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.0.0.1:32916)
0928 14:53:42.202686 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.202700 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.202785 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.202935 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.202935 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.203110 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.203162 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.203163 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.204205 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.204206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.204770 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.204772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.205283 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.205283 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.205340 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.205885 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.205891 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.206227 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.207999 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.208000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.208001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.208185 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.208395 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.208395 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.208396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.208448 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.208719 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.208719 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.208761 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.208815 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.208902 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":75,"threads_started":1}
I20260928 14:53:42.209045 27126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.202708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32916 (local address 127.20.116.190:35947)
0928 14:53:42.202788 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.202791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.202828 (+ 37us) server_negotiation.cc:407] Connection header received
0928 14:53:42.202859 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.202861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.202872 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.202931 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.203239 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.204093 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.204879 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.205189 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.205278 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.206320 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.206321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.206322 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.206329 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.207902 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.208260 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.208261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.208261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.208327 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.208511 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.208512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.208512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.208653 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.208724 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.208855 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.208949 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:53:42.209385 27106 tablet_copy_service.cc:140] P 2947c96f35da45d4b4bb59aeda6a1e0d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 51131b65b43344d59beff206ed38e6bd ({username='oryx', principal='oryx/127.20.116.189@KRBTEST.COM'} at 127.0.0.1:32916)
I20260928 14:53:42.209489 27106 tablet_copy_service.cc:161] P 2947c96f35da45d4b4bb59aeda6a1e0d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 51131b65b43344d59beff206ed38e6bd at {username='oryx', principal='oryx/127.20.116.189@KRBTEST.COM'} at 127.0.0.1:32916: session id = 51131b65b43344d59beff206ed38e6bd-00000000000000000000000000000000
I20260928 14:53:42.210402 27106 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 14:53:42.211177 27553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 14:53:42.212211 27553 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd: tablet copy: Starting download of 0 data blocks...
I20260928 14:53:42.212376 27553 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd: tablet copy: Starting download of 1 WAL segments...
I20260928 14:53:42.213268 27553 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 14:53:42.214130 27106 tablet_copy_service.cc:342] P 2947c96f35da45d4b4bb59aeda6a1e0d: Request end of tablet copy session 51131b65b43344d59beff206ed38e6bd-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.116.189@KRBTEST.COM'} at 127.0.0.1:32916
I20260928 14:53:42.214190 27106 tablet_copy_service.cc:433] P 2947c96f35da45d4b4bb59aeda6a1e0d: ending tablet copy session 51131b65b43344d59beff206ed38e6bd-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 51131b65b43344d59beff206ed38e6bd
I20260928 14:53:42.216701 27553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:42.218006 27660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:42.218006 27659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:42.218226 27662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:42.218323 27553 server_base.cc:1034] running on GCE node
I20260928 14:53:42.218493 27553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:42.218658 27553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:42.219789 27553 hybrid_clock.cc:648] HybridClock initialized: now 1790607222219777 us; error 31 us; skew 500 ppm
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607222, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:42.221160 27553 init.cc:377] Logged in from keytab as oryx/127.20.116.189@KRBTEST.COM (short username oryx)
I20260928 14:53:42.221457 27553 webserver.cc:460] Webserver started at http://127.20.116.189:42249/ using document root <none> and password file <none>
I20260928 14:53:42.221585 27553 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:42.221638 27553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:42.222278 27553 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:53:42.222996 27669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:42.223160 27553 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:42.223212 27553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/wal
uuid: "51131b65b43344d59beff206ed38e6bd"
format_stamp: "Formatted at 2026-09-28 14:53:42 on dist-test-slave-pxl6"
server_key: "db18a5bcc76910a06e469a156299535a"
server_key_iv: "a97a74443e19da2e22a9e6ab9785d050"
server_key_version: "encryptionkey@0"
I20260928 14:53:42.223284 27553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:42.241683 27553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:42.243918 27553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:42.247186 27553 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.189:37691
I20260928 14:53:42.247445 27721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.189:37691 every 8 connection(s)
I20260928 14:53:42.247568 27553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-1/data/info.pb
I20260928 14:53:42.248554 27722 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:53:42.249130 27722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd: Bootstrap starting.
I20260928 14:53:42.252499 27722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:53:42.252776 27722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd: Bootstrap complete.
I20260928 14:53:42.253067 27722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true } }
I20260928 14:53:42.253162 27722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 51131b65b43344d59beff206ed38e6bd, State: Initialized, Role: LEARNER
I20260928 14:53:42.253212 27722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true } }
I20260928 14:53:42.253425 27722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:42.253441 27725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true } } }
I20260928 14:53:42.253573 27725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [sys.catalog]: This master's current role is: LEARNER
I20260928 14:53:42.255075 27736 catalog_manager.cc:1259] Loaded cluster ID: b4b97e5e908a4355897c5dbba5724d5a
I20260928 14:53:42.255183 27736 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd: loading cluster ID for follower catalog manager: success
I20260928 14:53:42.256364 27736 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd: acquiring CA information for follower catalog manager: success
I20260928 14:53:42.256680 27736 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:53:42.300375 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.293564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33194 (local address 127.20.116.189:37691)
0928 14:53:42.293733 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.293735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.293750 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:53:42.293773 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.293774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.293783 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.293861 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.294444 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.295214 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.296070 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.296304 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.296385 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.297675 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.297676 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.297677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.297686 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.299176 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.299540 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.299541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.299542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.299606 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.299813 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.299814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.299815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.299934 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.300040 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.300150 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.300240 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for oryx/127.20.116.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.116.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.116.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.188.keytab.
Entry for principal oryx/127.20.116.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.188.keytab.
I20260928 14:53:42.327957 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.188:41705
--webserver_interface=127.20.116.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.188.keytab
--principal=oryx/127.20.116.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:35947,127.20.116.189:37691,127.20.116.188:41705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:42.414222 27740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:42.414428 27740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:42.414450 27740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:42.415659 27740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:53:42.415709 27740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:42.415723 27740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:42.415736 27740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:53:42.415748 27740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:53:42.417318 27740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:35947,127.20.116.189:37691,127.20.116.188:41705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.188.keytab
--principal=oryx/127.20.116.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.188:41705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.116.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:42.417582 27740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:42.417840 27740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:42.423030 27745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:42.423123 27746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:42.423242 27748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:42.423451 27740 server_base.cc:1034] running on GCE node
I20260928 14:53:42.423754 27740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:42.424094 27740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:42.425273 27740 hybrid_clock.cc:648] HybridClock initialized: now 1790607222425255 us; error 31 us; skew 500 ppm
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607222, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:42.427927 27740 init.cc:377] Logged in from keytab as oryx/127.20.116.188@KRBTEST.COM (short username oryx)
I20260928 14:53:42.428406 27740 webserver.cc:460] Webserver started at http://127.20.116.188:45743/ using document root <none> and password file <none>
I20260928 14:53:42.428627 27740 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:42.428669 27740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:42.428761 27740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:42.429889 27740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/data/instance:
uuid: "62ebad5801fc4460bfc70a394730e0b1"
format_stamp: "Formatted at 2026-09-28 14:53:42 on dist-test-slave-pxl6"
server_key: "4497afdc501f7128d8842f8436ff4e6c"
server_key_iv: "72cd56752b0d14146a454921f61836d0"
server_key_version: "encryptionkey@0"
I20260928 14:53:42.430253 27740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/wal/instance:
uuid: "62ebad5801fc4460bfc70a394730e0b1"
format_stamp: "Formatted at 2026-09-28 14:53:42 on dist-test-slave-pxl6"
server_key: "4497afdc501f7128d8842f8436ff4e6c"
server_key_iv: "72cd56752b0d14146a454921f61836d0"
server_key_version: "encryptionkey@0"
I20260928 14:53:42.431715 27740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:53:42.432512 27755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:42.432725 27740 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:42.432813 27740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/wal
uuid: "62ebad5801fc4460bfc70a394730e0b1"
format_stamp: "Formatted at 2026-09-28 14:53:42 on dist-test-slave-pxl6"
server_key: "4497afdc501f7128d8842f8436ff4e6c"
server_key_iv: "72cd56752b0d14146a454921f61836d0"
server_key_version: "encryptionkey@0"
I20260928 14:53:42.432870 27740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:42.443810 27740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:42.445771 27740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:42.445887 27740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:42.450126 27740 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.188:41705
I20260928 14:53:42.450160 27807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.188:41705 every 8 connection(s)
I20260928 14:53:42.450646 27740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/data/info.pb
I20260928 14:53:42.451716 27808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:42.453504 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 27740
I20260928 14:53:42.453393 27808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607222, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.188@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.116.188@KRBTEST.COM
I20260928 14:53:42.464509 27126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.454560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32918 (local address 127.20.116.190:35947)
0928 14:53:42.454609 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.454611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.454622 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:53:42.455233 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.455234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.455249 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.455303 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.456108 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.456950 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.458171 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.458477 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.458570 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.461193 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.461195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.461195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.461204 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.462875 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.463474 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.463475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.463475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.463529 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.463814 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.463815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.463815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.463922 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.463972 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.464295 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.464379 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 14:53:42.464668 27810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.454207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.0.0.1:32918)
0928 14:53:42.454459 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.454506 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.455113 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.455342 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.455345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.455771 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.456012 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.456019 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.457045 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.457046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.458069 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.458072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.458617 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.458618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.458740 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.459448 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.459459 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.461101 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.462958 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.462965 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.462968 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.463382 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.463608 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.463609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.463609 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.463743 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.463987 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.463993 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.464157 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.464241 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.464425 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":110,"threads_started":1}
I20260928 14:53:42.465242 27808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:53:42.465317 27811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.454209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42328 (local address 127.20.116.188:41705)
0928 14:53:42.454618 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.454622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.454656 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:53:42.455118 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.455134 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.455162 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.455291 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.456041 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.457257 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.458256 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.458730 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.458786 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.461915 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.461922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.461923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.461930 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.463936 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.464307 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.464310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.464310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.464371 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.464584 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.464585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.464585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.464751 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.464896 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.465014 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.465106 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":106,"threads_started":1}
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607222, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.188@KRBTEST.COM for oryx/127.20.116.189@KRBTEST.COM
I20260928 14:53:42.471894 27810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.465556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:37691 (local address 127.0.0.1:33206)
0928 14:53:42.465701 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.465710 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.465741 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.466085 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.466086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.466202 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.466235 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.466237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.466990 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.466991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.467492 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.467494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.467905 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.467906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.467950 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.468371 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.468376 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.469581 (+ 1205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.471034 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.471035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.471035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.471204 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.471385 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.471386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.471386 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.471435 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.471667 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.471668 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.471704 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.471756 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.471825 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260928 14:53:42.471903 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.465884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33206 (local address 127.20.116.189:37691)
0928 14:53:42.465955 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.465957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.465966 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:53:42.465995 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.465996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.466010 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.466067 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.466306 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.466917 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.467613 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.467832 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.467895 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.469665 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.469667 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.469667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.469674 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.470962 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.471271 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.471272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.471273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.471326 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.471489 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.471489 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.471489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.471609 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.471661 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.471775 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.471842 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:53:42.472605 27808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607222, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.188@KRBTEST.COM for oryx/127.20.116.188@KRBTEST.COM
I20260928 14:53:42.478979 27810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.472895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:41705 (local address 127.0.0.1:42336)
0928 14:53:42.472959 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.472970 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.473017 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.473125 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.473125 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.473230 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.473259 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.473260 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.473844 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.473844 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.474310 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.474311 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.474775 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.474776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.474815 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.475209 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.475213 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.476404 (+ 1191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.478034 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.478035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.478035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.478194 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.478375 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.478376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.478376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.478424 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.478738 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.478739 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.478775 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.478831 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.478910 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 14:53:42.479074 27811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.472933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42336 (local address 127.20.116.188:41705)
0928 14:53:42.472979 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.472981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.473019 (+ 38us) server_negotiation.cc:407] Connection header received
0928 14:53:42.473049 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.473050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.473059 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.473103 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.473317 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.473775 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.474392 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.474707 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.474770 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.476482 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.476484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.476484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.476490 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.477953 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.478261 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.478262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.478262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.478314 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.478479 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.478479 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.478479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.478678 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.478734 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.478862 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.478957 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":28}
I20260928 14:53:42.480419 27808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1: Bootstrap starting.
I20260928 14:53:42.481235 27808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:42.481472 27808 log.cc:824] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:42.482231 27808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1: No bootstrap required, opened a new log
I20260928 14:53:42.483774 27808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } }
I20260928 14:53:42.483907 27808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:42.483932 27808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62ebad5801fc4460bfc70a394730e0b1, State: Initialized, Role: FOLLOWER
I20260928 14:53:42.484062 27808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } }
I20260928 14:53:42.484357 27813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } } }
I20260928 14:53:42.484431 27808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:42.484447 27813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 14:53:42.486200 27824 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:53:42.486266 27824 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:53:42.488770 27740 master_runner.cc:428] Detected that this master 62ebad5801fc4460bfc70a394730e0b1 is joining an existing cluster
I20260928 14:53:42.488822 27740 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.116.188:41705
I20260928 14:53:42.497192 27831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.489953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.0.0.1:32930)
0928 14:53:42.490086 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.490096 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.490173 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.490356 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.490357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.490677 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.490748 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.490750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.491637 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.491638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.492564 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.492567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.493389 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.493390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.493458 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.493820 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.493825 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.494247 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.496099 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.496100 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.496100 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.496278 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.496492 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.496493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.496494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.496565 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.496837 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.496838 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.496896 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.496964 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.497073 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":28,"thread_start_us":75,"threads_started":1}
I20260928 14:53:42.497869 27126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.489885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32930 (local address 127.20.116.190:35947)
0928 14:53:42.489947 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.489950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.490148 (+ 198us) server_negotiation.cc:407] Connection header received
0928 14:53:42.490247 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.490248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.490259 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.490290 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.490829 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.491552 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.492664 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.493302 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.493377 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.494373 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.494375 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.494375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.494383 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.496013 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.496361 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.496362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.496363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.496425 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.496645 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.496646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.496646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.496768 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.496846 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.497320 (+ 474us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.497412 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:53:42.500897 27832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.490461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:41705 (local address 127.0.0.1:42338)
0928 14:53:42.494631 (+ 4170us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.494642 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.494713 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.494858 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.494859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.495042 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.495115 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.495117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.495997 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.495997 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.496852 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.496855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.497336 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.497337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.497409 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.497864 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.497870 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.498203 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.499841 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.499841 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.499842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.500005 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.500206 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.500207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.500207 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.500260 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.500504 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.500505 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.500554 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.500653 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.500753 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4149,"thread_start_us":154,"threads_started":1}
I20260928 14:53:42.500901 27811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.490381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42338 (local address 127.20.116.188:41705)
0928 14:53:42.490416 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.490418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.494691 (+ 4273us) server_negotiation.cc:407] Connection header received
0928 14:53:42.494779 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.494780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.494791 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.494845 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.495202 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.495908 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.496934 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.497221 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.497289 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.498304 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.498305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.498305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.498312 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.499738 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.500081 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.500082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.500082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.500133 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.500331 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.500332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.500332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.500438 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.500513 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.500676 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.500803 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":18}
I20260928 14:53:42.514367 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.490059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33220 (local address 127.20.116.189:37691)
0928 14:53:42.490105 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.490106 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.490115 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:53:42.490156 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.490157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.490166 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.490213 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.490731 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.491506 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.492380 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.492598 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.492674 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.511047 (+ 18373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.511050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.511050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.511061 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.512895 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.513295 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.513297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.513297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.513371 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.513591 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.513593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.513593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.513726 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.513799 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.514185 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.514274 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:53:42.514377 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.489879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:37691 (local address 127.0.0.1:33220)
0928 14:53:42.490030 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.490043 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.490107 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.490216 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.490217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.490525 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.490651 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.490653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.491614 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.491616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.492288 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.492290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.492747 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.492748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.492808 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.493310 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.493316 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.510897 (+ 17581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.512984 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.512985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.512985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.513193 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.513447 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.513447 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.513448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.513515 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.513811 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.513813 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.513866 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.514138 (+ 272us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.514255 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":50,"thread_start_us":99,"threads_started":1}
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607221, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.190@KRBTEST.COM for oryx/127.20.116.188@KRBTEST.COM
I20260928 14:53:42.516685 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 27149
I20260928 14:53:42.519666 27646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.512171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:41705 (local address 127.0.0.1:42340)
0928 14:53:42.512221 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.512234 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.512282 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.512452 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.512453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.512632 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.512706 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.512709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.513649 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.513650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.514443 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.514447 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.514840 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.514841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.514907 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.515306 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.515311 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.516756 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.518621 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.518622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.518622 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.518831 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.519020 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.519021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.519021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.519074 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.519334 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.519335 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.519378 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.519439 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.519548 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 14:53:42.520023 27811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.512268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42340 (local address 127.20.116.188:41705)
0928 14:53:42.512309 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.512311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.512321 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:53:42.512356 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.512357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.512368 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.512429 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.512797 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.513559 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.514558 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.514764 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.514843 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.516854 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.516856 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.516856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.516864 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.518495 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.518907 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.518908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.518908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.518957 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.519136 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.519136 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.519137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.519255 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.519309 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.519862 (+ 553us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.519937 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:53:42.520443 27086 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.116.188:41705: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1"
member_type: NON_VOTER
last_known_addr {
host: "127.20.116.188"
port: 41705
}
attrs {
promote: true
}
}
dest_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d"
cas_config_opid_index: 5
I20260928 14:53:42.520728 27086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: NON_VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: true } }
I20260928 14:53:42.521163 27648 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 62ebad5801fc4460bfc70a394730e0b1 (127.20.116.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: NON_VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: true } } }
I20260928 14:53:42.521377 27782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:42.522253 27648 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.116.188:41705
I20260928 14:53:42.522324 27782 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 1 FOLLOWER]: Refusing update from remote peer 2947c96f35da45d4b4bb59aeda6a1e0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 14:53:42.522337 27648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: NON_VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: true } } }
I20260928 14:53:42.522414 27648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:42.522851 27647 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: NON_VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:42.523284 27740 master.cc:537] Master@127.20.116.188:41705 shutting down...
I20260928 14:53:42.524214 27782 raft_consensus.cc:858] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: NON_VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: true } }
W20260928 14:53:42.524286 27782 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: NON_VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: true } }
I20260928 14:53:42.524351 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:35257
--local_ip_for_outbound_sockets=127.20.116.129
--tserver_master_addrs=127.20.116.190:35947
--webserver_port=44905
--webserver_interface=127.20.116.129
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.129.keytab
--principal=oryx/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.116.190:35947,127.20.116.189:37691
--tserver_master_addrs=127.20.116.190:35947,127.20.116.189:37691,127.20.116.188:41705 with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:53:42.524739 27782 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 51131b65b43344d59beff206ed38e6bd (127.20.116.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true } } }
I20260928 14:53:42.525686 27813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2947c96f35da45d4b4bb59aeda6a1e0d. Latest consensus state: current_term: 1 leader_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true } } }
I20260928 14:53:42.525761 27813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 14:53:42.525792 27813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true } } }
I20260928 14:53:42.525826 27813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 14:53:42.525852 27813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true } } }
I20260928 14:53:42.525885 27813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 14:53:42.527607 27833 mvcc.cc:205] Tried to move back new op lower bound from 7334327179071574016 to 7334327178988961792. Current Snapshot: MvccSnapshot[applied={T|T < 7334327178999783424}]
I20260928 14:53:42.528959 27646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.522699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:37691 (local address 127.0.0.1:33224)
0928 14:53:42.522758 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.522769 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.522810 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.525231 (+ 2421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.525233 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.525235 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:42.525282 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.525284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.526307 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.526308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.527613 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.527617 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.528650 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.528651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.528728 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.528758 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.528813 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.528859 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 14:53:42.529464 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.522804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33224 (local address 127.20.116.189:37691)
0928 14:53:42.525036 (+ 2232us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.525038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.525048 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:53:42.525076 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.525077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.525101 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.525166 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:42.525351 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.526091 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.527768 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.528453 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.528524 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.528614 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.529309 (+ 695us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.529358 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2206}
I20260928 14:53:42.529989 27696 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 1 LEARNER]: Refusing update from remote peer 2947c96f35da45d4b4bb59aeda6a1e0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 14:53:42.530239 27648 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 14:53:42.530941 27696 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 62ebad5801fc4460bfc70a394730e0b1 (127.20.116.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: NON_VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: true } } }
I20260928 14:53:42.531423 27648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: NON_VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: true } } }
I20260928 14:53:42.531523 27648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: This master's current role is: LEADER
W20260928 14:53:42.532133 27071 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d -> Peer 62ebad5801fc4460bfc70a394730e0b1 (127.20.116.188:41705): Couldn't send request to peer 62ebad5801fc4460bfc70a394730e0b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 14:53:42.532280 27725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2947c96f35da45d4b4bb59aeda6a1e0d. Latest consensus state: current_term: 1 leader_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: NON_VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: true } } }
I20260928 14:53:42.532368 27725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [sys.catalog]: This master's current role is: LEARNER
I20260928 14:53:42.532413 27725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: NON_VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: true } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: NON_VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: true } } }
I20260928 14:53:42.532469 27725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [sys.catalog]: This master's current role is: LEARNER
I20260928 14:53:42.533165 27740 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260928 14:53:42.533293 27740 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260928 14:53:42.533339 27740 tablet_replica.cc:330] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1: stopping tablet replica
I20260928 14:53:42.546980 27647 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d: attempting to promote NON_VOTER 51131b65b43344d59beff206ed38e6bd to VOTER
I20260928 14:53:42.547168 27740 master.cc:552] Master@127.20.116.188:41705 shutdown complete.
I20260928 14:53:42.547215 27740 master_runner.cc:305] Clearing existing system tablet
I20260928 14:53:42.547345 27740 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:42.547382 27740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:42.547398 27647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: NON_VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: true } }
I20260928 14:53:42.548018 27696 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 1 LEARNER]: Refusing update from remote peer 2947c96f35da45d4b4bb59aeda6a1e0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 14:53:42.548200 27648 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 14:53:42.548569 27646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.548427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:41705 (local address 127.0.0.1:42350)
0928 14:53:42.548477 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.548523 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:41705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 14:53:42.548834 27648 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 51131b65b43344d59beff206ed38e6bd (127.20.116.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: NON_VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: true } } }
I20260928 14:53:42.548920 27646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.548852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:41705 (local address 127.0.0.1:42354)
0928 14:53:42.548881 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.548901 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:41705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 14:53:42.549077 27071 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d -> Peer 62ebad5801fc4460bfc70a394730e0b1 (127.20.116.188:41705): Couldn't send request to peer 62ebad5801fc4460bfc70a394730e0b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.116.188:41705: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 14:53:42.549315 27740 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.000s
I20260928 14:53:42.549744 27648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: NON_VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: true } } }
I20260928 14:53:42.549825 27648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:42.549875 27648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: NON_VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: true } } }
I20260928 14:53:42.549933 27648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:42.550156 27845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:42.550287 27740 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:42.550225 27696 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 51131b65b43344d59beff206ed38e6bd (127.20.116.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: NON_VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: true } } }
I20260928 14:53:42.550330 27740 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:53:42.550351 27740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/wal
uuid: "62ebad5801fc4460bfc70a394730e0b1"
format_stamp: "Formatted at 2026-09-28 14:53:42 on dist-test-slave-pxl6"
server_key: "4497afdc501f7128d8842f8436ff4e6c"
server_key_iv: "72cd56752b0d14146a454921f61836d0"
server_key_version: "encryptionkey@0"
I20260928 14:53:42.551120 27740 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:42.551204 27842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: NON_VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: true } } }
I20260928 14:53:42.551298 27842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:53:42.552501 27740 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 14:53:42.552593 27740 log.cc:1196] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 14:53:42.552829 27740 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1: Deleting consensus metadata
I20260928 14:53:42.552968 27740 master_runner.cc:315] Copying system tablet from 127.20.116.190:35947
I20260928 14:53:42.553601 27740 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.116.190:35947
I20260928 14:53:42.561197 27851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.553900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.0.0.1:32942)
0928 14:53:42.554233 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.554244 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.554312 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.554478 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.554478 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.554762 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.554824 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.554826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.555859 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.555861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.556711 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.556714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.557199 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.557201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.557260 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.557802 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.557807 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.558189 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.560043 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.560044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.560044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.560265 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.560488 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.560489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.560489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.560554 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.560860 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.560861 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.560913 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.560985 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.561096 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":63,"threads_started":1}
I20260928 14:53:42.561197 27126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.553959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.20.116.190:35947)
0928 14:53:42.553999 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.554001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.554312 (+ 311us) server_negotiation.cc:407] Connection header received
0928 14:53:42.554381 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.554382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.554395 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.554468 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.554915 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.555746 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.556837 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.557115 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.557190 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.558286 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.558288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.558288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.558296 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.559931 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.560348 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.560349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.560349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.560414 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.560637 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.560638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.560638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.560788 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.560859 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.561028 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.561111 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:53:42.561898 27106 tablet_copy_service.cc:140] P 2947c96f35da45d4b4bb59aeda6a1e0d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 62ebad5801fc4460bfc70a394730e0b1 ({username='oryx', principal='oryx/127.20.116.188@KRBTEST.COM'} at 127.0.0.1:32942)
I20260928 14:53:42.561969 27106 tablet_copy_service.cc:161] P 2947c96f35da45d4b4bb59aeda6a1e0d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 62ebad5801fc4460bfc70a394730e0b1 at {username='oryx', principal='oryx/127.20.116.188@KRBTEST.COM'} at 127.0.0.1:32942: session id = 62ebad5801fc4460bfc70a394730e0b1-00000000000000000000000000000000
I20260928 14:53:42.562656 27106 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 14:53:42.563155 27740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 14:53:42.564419 27740 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1: tablet copy: Starting download of 0 data blocks...
I20260928 14:53:42.564576 27740 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1: tablet copy: Starting download of 1 WAL segments...
I20260928 14:53:42.565307 27740 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 14:53:42.566251 27106 tablet_copy_service.cc:342] P 2947c96f35da45d4b4bb59aeda6a1e0d: Request end of tablet copy session 62ebad5801fc4460bfc70a394730e0b1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.116.188@KRBTEST.COM'} at 127.0.0.1:32942
I20260928 14:53:42.566313 27106 tablet_copy_service.cc:433] P 2947c96f35da45d4b4bb59aeda6a1e0d: ending tablet copy session 62ebad5801fc4460bfc70a394730e0b1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 62ebad5801fc4460bfc70a394730e0b1
I20260928 14:53:42.568536 27740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:42.569701 27852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:42.569741 27855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:42.570253 27853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:42.570408 27740 server_base.cc:1034] running on GCE node
I20260928 14:53:42.570514 27740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:42.570686 27740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:42.578051 27740 hybrid_clock.cc:648] HybridClock initialized: now 1790607222578025 us; error 46 us; skew 500 ppm
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607222, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:42.579698 27740 init.cc:377] Logged in from keytab as oryx/127.20.116.188@KRBTEST.COM (short username oryx)
I20260928 14:53:42.580010 27740 webserver.cc:460] Webserver started at http://127.20.116.188:33717/ using document root <none> and password file <none>
I20260928 14:53:42.580163 27740 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:42.580202 27740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:42.581009 27740 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:53:42.581652 27862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:42.581816 27740 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:53:42.581879 27740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/wal
uuid: "62ebad5801fc4460bfc70a394730e0b1"
format_stamp: "Formatted at 2026-09-28 14:53:42 on dist-test-slave-pxl6"
server_key: "4497afdc501f7128d8842f8436ff4e6c"
server_key_iv: "72cd56752b0d14146a454921f61836d0"
server_key_version: "encryptionkey@0"
I20260928 14:53:42.581954 27740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:42.607245 27740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:42.609694 27740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:42.613389 27740 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.188:41705
I20260928 14:53:42.613508 27914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.188:41705 every 8 connection(s)
I20260928 14:53:42.613765 27740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/master-2/data/info.pb
I20260928 14:53:42.614744 27915 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:53:42.615278 27915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1: Bootstrap starting.
I20260928 14:53:42.617738 27915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:53:42.617950 27915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1: Bootstrap complete.
I20260928 14:53:42.618160 27915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: NON_VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: true } }
I20260928 14:53:42.618252 27915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 62ebad5801fc4460bfc70a394730e0b1, State: Initialized, Role: LEARNER
I20260928 14:53:42.618291 27915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: NON_VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: true } }
I20260928 14:53:42.618474 27915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:42.618485 27918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: NON_VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: true } } }
I20260928 14:53:42.618757 27918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [sys.catalog]: This master's current role is: LEARNER
I20260928 14:53:42.620155 27929 catalog_manager.cc:1259] Loaded cluster ID: b4b97e5e908a4355897c5dbba5724d5a
I20260928 14:53:42.620239 27929 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1: loading cluster ID for follower catalog manager: success
I20260928 14:53:42.621426 27929 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1: acquiring CA information for follower catalog manager: success
I20260928 14:53:42.621861 27929 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 14:53:42.633323 27836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:42.633610 27836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:42.633654 27836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:42.634995 27836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:42.635067 27836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:42.635119 27836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:53:42.636694 27836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.129.keytab
--principal=oryx/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:35257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=44905
--tserver_master_addrs=127.20.116.190:35947,127.20.116.189:37691,127.20.116.188:41705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:42.636973 27836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:42.637228 27836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:42.643151 27934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:42.643141 27935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:42.643388 27836 server_base.cc:1034] running on GCE node
W20260928 14:53:42.643141 27937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:42.643656 27836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:42.643875 27836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:42.645023 27836 hybrid_clock.cc:648] HybridClock initialized: now 1790607222645008 us; error 27 us; skew 500 ppm
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607222, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:42.647687 27836 init.cc:377] Logged in from keytab as oryx/127.20.116.129@KRBTEST.COM (short username oryx)
I20260928 14:53:42.648209 27836 webserver.cc:460] Webserver started at http://127.20.116.129:44905/ using document root <none> and password file <none>
I20260928 14:53:42.648402 27836 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:42.648452 27836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:42.649947 27836 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:42.650492 27944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:42.650683 27836 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:42.650774 27836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "dd1d5fa791f449278948c4abbb832e43"
format_stamp: "Formatted at 2026-09-28 14:53:41 on dist-test-slave-pxl6"
server_key: "109c903c742f51a56963d285d7fcd439"
server_key_iv: "a92324a2c5ad304a2a5de50e6910bddf"
server_key_version: "encryptionkey@0"
I20260928 14:53:42.651144 27836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:42.673188 27836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:42.675709 27836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:42.675866 27836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:42.676157 27836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:42.676513 27836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:42.676568 27836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:42.676605 27836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:42.676648 27836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:42.682801 27836 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:35257
I20260928 14:53:42.682888 28057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:35257 every 8 connection(s)
I20260928 14:53:42.683254 27836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607222, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607222, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.189@KRBTEST.COM
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607222, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.188@KRBTEST.COM
I20260928 14:53:42.692538 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 27836
I20260928 14:53:42.692711 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 27284
I20260928 14:53:42.695793 27126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.684326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:42361 (local address 127.20.116.190:35947)
0928 14:53:42.684376 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.684378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.684422 (+ 44us) server_negotiation.cc:407] Connection header received
0928 14:53:42.685532 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.685533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.685546 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.685580 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.686489 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.687287 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.688553 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.688851 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.688931 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.691652 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.691656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.691657 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.691672 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.693675 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.694500 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.694501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.694502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.694585 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.694923 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.694924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.694925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.695081 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.695145 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.695542 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.695639 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:53:42.695880 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.684136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:43277 (local address 127.20.116.189:37691)
0928 14:53:42.684188 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.684191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.684504 (+ 313us) server_negotiation.cc:407] Connection header received
0928 14:53:42.685531 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.685533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.685546 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.685580 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.686488 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.687361 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.688571 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.688839 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.688902 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.692041 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.692042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.692042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.692052 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.694461 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.694967 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.694968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.694969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.695025 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.695254 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.695255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.695255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.695407 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.695450 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.695686 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.695772 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:53:42.696368 28062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.684007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:37691 (local address 127.20.116.129:43277)
0928 14:53:42.684441 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.684451 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.685415 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.685701 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.685701 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.686167 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.686376 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.686382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.687450 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.687451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.688459 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.688462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.688956 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.688957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.689015 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.689700 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.689712 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.691941 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.694637 (+ 2696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.694638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.694639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.694874 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.695101 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.695102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.695102 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.695172 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.695513 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.695514 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.695567 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.695995 (+ 428us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.696158 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":202,"thread_start_us":93,"threads_started":1}
I20260928 14:53:42.696647 28065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.684314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:40679 (local address 127.20.116.188:41705)
0928 14:53:42.684421 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.684423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.684432 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:53:42.685513 (+ 1081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.685515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.685531 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.685602 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.686475 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.687196 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.688383 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.688683 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.688753 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.692556 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.692558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.692558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.692567 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.694877 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.695292 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.695294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.695294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.695347 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.695543 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.695543 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.695544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.695681 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.695740 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.696425 (+ 685us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.696534 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":58,"threads_started":1}
I20260928 14:53:42.696368 28063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.684007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:41705 (local address 127.20.116.129:40679)
0928 14:53:42.684331 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.684343 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.685393 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.685674 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.685678 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.686161 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.686375 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.686379 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.687312 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.687313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.688288 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.688290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.688774 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.688775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.688936 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.689700 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.689712 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.692476 (+ 2764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.694958 (+ 2482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.694959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.694959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.695218 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.695416 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.695417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.695417 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.695475 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.695767 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.695768 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.695802 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.695874 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.696109 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":77,"thread_start_us":110,"threads_started":1}
I20260928 14:53:42.696902 28060 heartbeater.cc:347] Connected to a master server at 127.20.116.189:37691
I20260928 14:53:42.696985 28060 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:42.697198 28060 heartbeater.cc:511] Master 127.20.116.189:37691 requested a full tablet report, sending...
I20260928 14:53:42.697324 28064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.684006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.129:42361)
0928 14:53:42.684291 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.684335 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.685415 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.685693 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.685694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.686160 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.686376 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.686382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.687383 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.687385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.688461 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.688462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.688932 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.688933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.689004 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.689753 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.689758 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.691513 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.693766 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.693773 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.693776 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.694404 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.694662 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.694663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.694663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.694844 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.695203 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.695209 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.695388 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.696404 (+ 1016us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.697129 (+ 725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":71,"thread_start_us":113,"threads_started":1}
I20260928 14:53:42.697669 28058 heartbeater.cc:347] Connected to a master server at 127.20.116.188:41705
I20260928 14:53:42.697723 28058 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:42.697804 28058 heartbeater.cc:511] Master 127.20.116.188:41705 requested a full tablet report, sending...
I20260928 14:53:42.697830 27686 ts_manager.cc:194] Registered new tserver with Master: dd1d5fa791f449278948c4abbb832e43 (127.20.116.129:35257)
I20260928 14:53:42.698195 27879 ts_manager.cc:194] Registered new tserver with Master: dd1d5fa791f449278948c4abbb832e43 (127.20.116.129:35257)
I20260928 14:53:42.699129 28059 heartbeater.cc:347] Connected to a master server at 127.20.116.190:35947
I20260928 14:53:42.699187 28059 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:42.699252 28059 heartbeater.cc:511] Master 127.20.116.190:35947 requested a full tablet report, sending...
I20260928 14:53:42.699515 27086 ts_manager.cc:194] Re-registered known tserver with Master: dd1d5fa791f449278948c4abbb832e43 (127.20.116.129:35257)
I20260928 14:53:42.699909 27086 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:42361
I20260928 14:53:42.701206 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:35679
--local_ip_for_outbound_sockets=127.20.116.130
--tserver_master_addrs=127.20.116.190:35947
--webserver_port=38111
--webserver_interface=127.20.116.130
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.130.keytab
--principal=oryx/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.116.190:35947,127.20.116.189:37691
--tserver_master_addrs=127.20.116.190:35947,127.20.116.189:37691,127.20.116.188:41705 with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:42.785948 28066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:42.786193 28066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:42.786231 28066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:42.787498 28066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:42.787577 28066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:42.787635 28066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:53:42.789223 28066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.130.keytab
--principal=oryx/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:35679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=38111
--tserver_master_addrs=127.20.116.190:35947,127.20.116.189:37691,127.20.116.188:41705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:42.789513 28066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:42.789776 28066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:42.795408 28071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:42.795410 28072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:42.795401 28074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:42.795456 28066 server_base.cc:1034] running on GCE node
I20260928 14:53:42.795892 28066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:42.796181 28066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:42.797366 28066 hybrid_clock.cc:648] HybridClock initialized: now 1790607222797355 us; error 38 us; skew 500 ppm
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607222, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:42.799993 28066 init.cc:377] Logged in from keytab as oryx/127.20.116.130@KRBTEST.COM (short username oryx)
I20260928 14:53:42.800452 28066 webserver.cc:460] Webserver started at http://127.20.116.130:38111/ using document root <none> and password file <none>
I20260928 14:53:42.800709 28066 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:42.800792 28066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:42.802313 28066 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:53:42.803031 28081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:42.803239 28066 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:53:42.803340 28066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "d18126ddd7c840049afba377de0fa029"
format_stamp: "Formatted at 2026-09-28 14:53:41 on dist-test-slave-pxl6"
server_key: "43448885792c1878c26edf00d206550b"
server_key_iv: "f26878272862b69c11a35d78d3cc68a1"
server_key_version: "encryptionkey@0"
I20260928 14:53:42.803709 28066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:42.810845 28066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:42.812803 28066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:42.812978 28066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:42.813231 28066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:42.813560 28066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:42.813616 28066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:42.813659 28066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:42.813690 28066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:42.820179 28066 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:35679
I20260928 14:53:42.820274 28194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:35679 every 8 connection(s)
I20260928 14:53:42.820676 28066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:53:42.826040 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 28066
I20260928 14:53:42.826213 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 27418
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607222, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.189@KRBTEST.COM
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607222, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.188@KRBTEST.COM
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607222, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM
I20260928 14:53:42.833392 28065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.821547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:33047 (local address 127.20.116.188:41705)
0928 14:53:42.821596 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.821599 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.821861 (+ 262us) server_negotiation.cc:407] Connection header received
0928 14:53:42.822448 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.822450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.822466 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.822522 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.823599 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.824229 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.825302 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.825578 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.825645 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.829888 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.829889 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.829889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.829898 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.831626 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.832124 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.832125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.832125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.832182 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.832476 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.832477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.832478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.832627 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.832670 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.833148 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.833261 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:53:42.833448 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.821858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:45675 (local address 127.20.116.189:37691)
0928 14:53:42.821908 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.821911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.821988 (+ 77us) server_negotiation.cc:407] Connection header received
0928 14:53:42.822505 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.822506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.822546 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.822583 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.823620 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.824386 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.825538 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.825826 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.825888 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.829556 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.829558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.829558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.829565 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.831547 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.832448 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.832449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.832449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.832505 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.832735 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.832736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.832737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.832902 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.832992 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.833258 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.833337 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:53:42.833957 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:42409
--local_ip_for_outbound_sockets=127.20.116.131
--tserver_master_addrs=127.20.116.190:35947
--webserver_port=34011
--webserver_interface=127.20.116.131
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.131.keytab
--principal=oryx/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.116.190:35947,127.20.116.189:37691
--tserver_master_addrs=127.20.116.190:35947,127.20.116.189:37691,127.20.116.188:41705 with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:53:42.833968 28200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.821479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:41705 (local address 127.20.116.130:33047)
0928 14:53:42.821718 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.821762 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.822340 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.822622 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.822626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.823231 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.823500 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.823505 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.824329 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.824330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.825198 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.825210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.825667 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.825668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.825823 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.826629 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.826640 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.829791 (+ 3151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.831717 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.831718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.831719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.832039 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.832269 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.832270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.832270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.832406 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.832716 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.832722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.832950 (+ 228us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.833521 (+ 571us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.833715 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260928 14:53:42.834822 28195 heartbeater.cc:347] Connected to a master server at 127.20.116.188:41705
I20260928 14:53:42.834952 28195 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:42.835134 28199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.821479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:37691 (local address 127.20.116.130:45675)
0928 14:53:42.821934 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.821943 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.822382 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.822689 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.822690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.823212 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.823501 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.823508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.824482 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.824483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.825442 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.825443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.825943 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.825944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.826011 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.826807 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.826811 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.829450 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.831663 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.831671 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.831674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.832310 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.832593 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.832594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.832594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.832656 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.833066 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.833067 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.833123 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.833777 (+ 654us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.833918 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":244,"thread_start_us":61,"threads_started":1}
I20260928 14:53:42.835182 28195 heartbeater.cc:511] Master 127.20.116.188:41705 requested a full tablet report, sending...
I20260928 14:53:42.835834 27879 ts_manager.cc:194] Registered new tserver with Master: d18126ddd7c840049afba377de0fa029 (127.20.116.130:35679)
I20260928 14:53:42.835971 28197 heartbeater.cc:347] Connected to a master server at 127.20.116.189:37691
I20260928 14:53:42.836021 28197 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:42.836097 28197 heartbeater.cc:511] Master 127.20.116.189:37691 requested a full tablet report, sending...
I20260928 14:53:42.836414 27686 ts_manager.cc:194] Registered new tserver with Master: d18126ddd7c840049afba377de0fa029 (127.20.116.130:35679)
I20260928 14:53:42.837268 28201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.821480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.130:55807)
0928 14:53:42.821770 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.821784 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.822379 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.822667 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.822667 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.823212 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.823501 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.823508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.824483 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.824483 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.825420 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.825423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.831685 (+ 6262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.831686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.831776 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.832196 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.832202 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.833952 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.836274 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.836275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.836275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.836448 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.836718 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.836719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.836719 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.836768 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.837027 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.837028 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.837060 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.837113 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.837188 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":90,"thread_start_us":82,"threads_started":1}
I20260928 14:53:42.837371 27126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.821744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:55807 (local address 127.20.116.190:35947)
0928 14:53:42.821804 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.821807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.821928 (+ 121us) server_negotiation.cc:407] Connection header received
0928 14:53:42.822505 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.822514 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.822562 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.822602 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.823629 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.824382 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.825505 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.825777 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.825859 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.834036 (+ 8177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.834038 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.834038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.834046 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.836147 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.836584 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.836585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.836585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.836653 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.836821 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.836822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.836822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.836966 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.837031 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.837141 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.837233 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 14:53:42.837693 28196 heartbeater.cc:347] Connected to a master server at 127.20.116.190:35947
I20260928 14:53:42.837742 28196 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:42.837805 28196 heartbeater.cc:511] Master 127.20.116.190:35947 requested a full tablet report, sending...
I20260928 14:53:42.838083 27086 ts_manager.cc:194] Re-registered known tserver with Master: d18126ddd7c840049afba377de0fa029 (127.20.116.130:35679)
I20260928 14:53:42.838692 27086 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:55807
W20260928 14:53:42.918324 28202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:42.918582 28202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:42.918648 28202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:42.919968 28202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:42.920032 28202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:42.920064 28202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:53:42.921659 28202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.131.keytab
--principal=oryx/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:42409
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=34011
--tserver_master_addrs=127.20.116.190:35947,127.20.116.189:37691,127.20.116.188:41705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:42.921890 28202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:42.922079 28202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:42.927260 28210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:42.927254 28207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:42.927271 28208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:42.927278 28202 server_base.cc:1034] running on GCE node
I20260928 14:53:42.927749 28202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:42.927953 28202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:42.929129 28202 hybrid_clock.cc:648] HybridClock initialized: now 1790607222929122 us; error 39 us; skew 500 ppm
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607222, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:42.931658 28202 init.cc:377] Logged in from keytab as oryx/127.20.116.131@KRBTEST.COM (short username oryx)
I20260928 14:53:42.932135 28202 webserver.cc:460] Webserver started at http://127.20.116.131:34011/ using document root <none> and password file <none>
I20260928 14:53:42.932365 28202 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:42.932451 28202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:42.934129 28202 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:53:42.934836 28217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:42.935012 28202 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:53:42.935117 28202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "c0148ddeb6e042fbb5ad121e91de2860"
format_stamp: "Formatted at 2026-09-28 14:53:41 on dist-test-slave-pxl6"
server_key: "484d1eea8cb0bb6c4db3fac2a743e393"
server_key_iv: "53f1717e9eb5b2bf9e3fdd33578be44e"
server_key_version: "encryptionkey@0"
I20260928 14:53:42.935475 28202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:42.957540 28202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:42.959870 28202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:42.960048 28202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:42.960289 28202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:42.960609 28202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:42.960662 28202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:42.960711 28202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:42.960740 28202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:42.966753 28202 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:42409
I20260928 14:53:42.966811 28330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:42409 every 8 connection(s)
I20260928 14:53:42.967191 28202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:53:42.970170 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 28202
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607222, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.188@KRBTEST.COM
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607222, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.189@KRBTEST.COM
Sep 28 14:53:42 dist-test-slave-pxl6 krb5kdc[27041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607222, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM
I20260928 14:53:42.979997 28335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.967944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:37691 (local address 127.20.116.131:33829)
0928 14:53:42.968153 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.968202 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.968762 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.969021 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.969026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.969532 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.969792 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.969798 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.970833 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.970834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.973015 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.973018 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.973510 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.973511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.973576 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.973967 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.973973 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.975977 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.977991 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.977999 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.978001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.978343 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.978615 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.978616 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.978616 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.978780 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.979136 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.979142 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.979340 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.979453 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.979681 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":1166,"thread_start_us":85,"threads_started":1}
I20260928 14:53:42.980502 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.967955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:33829 (local address 127.20.116.189:37691)
0928 14:53:42.968198 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.968205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.968302 (+ 97us) server_negotiation.cc:407] Connection header received
0928 14:53:42.968887 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.968888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.968901 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.968936 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.969904 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.970714 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.973103 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.973394 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.973455 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.976093 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.976095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.976095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.976103 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.977893 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.978432 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.978433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.978434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.978506 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.978871 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.978872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.978873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.979024 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.979092 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.980302 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.980388 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260928 14:53:42.980942 28333 heartbeater.cc:347] Connected to a master server at 127.20.116.189:37691
I20260928 14:53:42.981040 28333 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:42.981269 28333 heartbeater.cc:511] Master 127.20.116.189:37691 requested a full tablet report, sending...
I20260928 14:53:42.981882 27686 ts_manager.cc:194] Registered new tserver with Master: c0148ddeb6e042fbb5ad121e91de2860 (127.20.116.131:42409)
I20260928 14:53:42.982628 28346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.971697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42358 (local address 127.20.116.188:41705)
0928 14:53:42.971809 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.971811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.971979 (+ 168us) server_negotiation.cc:407] Connection header received
0928 14:53:42.972002 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.972003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.972015 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.972070 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.974899 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.975684 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.976725 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.977035 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.977111 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.979335 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.979337 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.979337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.979344 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.981057 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.981516 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.981517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.981517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.981579 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.981780 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.981781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.981782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.981908 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.982028 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.982390 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.982488 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260928 14:53:42.983371 28336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.968058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:41705 (local address 127.20.116.131:55547)
0928 14:53:42.968275 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.968284 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.968764 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.969021 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.969026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.969533 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.969792 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.969798 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.970946 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.970947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.972120 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.972123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.972674 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.972675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.972813 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.973372 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.973382 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.976413 (+ 3031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.981825 (+ 5412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.981826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.981826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.982044 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.982380 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.982381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.982382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.982443 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.982984 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.982985 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.983034 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.983110 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.983236 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":95,"thread_start_us":84,"threads_started":1}
I20260928 14:53:42.983726 28065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.968122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:55547 (local address 127.20.116.188:41705)
0928 14:53:42.968215 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.968217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.968328 (+ 111us) server_negotiation.cc:407] Connection header received
0928 14:53:42.968887 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.968888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.968901 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.968936 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.969904 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.970696 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.972223 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.972543 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.972607 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.976527 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.976528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.976528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.976537 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.981742 (+ 5205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.982252 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.982253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.982253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.982308 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.982522 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.982523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.982523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.982662 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.982749 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.983559 (+ 810us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.983640 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:53:42.984023 28331 heartbeater.cc:347] Connected to a master server at 127.20.116.188:41705
I20260928 14:53:42.984120 28331 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:42.984202 28331 heartbeater.cc:511] Master 127.20.116.188:41705 requested a full tablet report, sending...
I20260928 14:53:42.984484 27879 ts_manager.cc:194] Registered new tserver with Master: c0148ddeb6e042fbb5ad121e91de2860 (127.20.116.131:42409)
I20260928 14:53:42.985494 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.971463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33226 (local address 127.20.116.189:37691)
0928 14:53:42.971574 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.971576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.977262 (+ 5686us) server_negotiation.cc:407] Connection header received
0928 14:53:42.977367 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.977369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.977382 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.977424 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.978099 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.978923 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.980000 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.980318 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.980360 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.982272 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.982273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.982274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.982281 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.984025 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.984587 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.984588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.984589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.984649 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.984860 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.984861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.984861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.984988 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.985101 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.985314 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.985392 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
I20260928 14:53:42.985703 27126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.967977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:58979 (local address 127.20.116.190:35947)
0928 14:53:42.968042 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.968045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.968460 (+ 415us) server_negotiation.cc:407] Connection header received
0928 14:53:42.968865 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.968866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.968879 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.968940 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.969892 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.970724 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.973381 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.973686 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.973980 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.977473 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.977475 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.977475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.977483 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.981584 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.984852 (+ 3268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.984853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.984853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.984915 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.985094 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.985095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.985095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.985236 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.985301 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.985489 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.985577 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 14:53:42.985797 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.971341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.20.116.190:35947)
0928 14:53:42.971494 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.971496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.971504 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:53:42.971590 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.971591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.971602 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.971661 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:42.972478 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.973212 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.974226 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.974580 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.974643 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.976213 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:42.976214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:42.976215 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:42.976222 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:42.984428 (+ 8206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.984922 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.984923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.984923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.984971 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:42.985157 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:42.985157 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:42.985158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:42.985257 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:42.985340 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.985523 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.985653 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260928 14:53:42.986617 28337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.967944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.131:58979)
0928 14:53:42.968388 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.968398 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.968774 (+ 376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.969320 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.969320 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.969524 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:42.969790 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.969795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.970836 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.970837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.972959 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.972961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.974061 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.974062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.974128 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.974554 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:42.974559 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:42.977355 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:42.984564 (+ 7209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.984565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.984565 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.984773 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.984989 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:42.984990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:42.984990 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:42.985047 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:42.985328 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:42.985329 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:42.985372 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.985801 (+ 429us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.985918 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":1483,"thread_start_us":83,"threads_started":1}
I20260928 14:53:42.986976 28332 heartbeater.cc:347] Connected to a master server at 127.20.116.190:35947
I20260928 14:53:42.987030 28332 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:42.987100 28332 heartbeater.cc:511] Master 127.20.116.190:35947 requested a full tablet report, sending...
I20260928 14:53:42.987366 27086 ts_manager.cc:194] Re-registered known tserver with Master: c0148ddeb6e042fbb5ad121e91de2860 (127.20.116.131:42409)
I20260928 14:53:42.987833 27086 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:58979
I20260928 14:53:42.989406 28065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.986323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42374 (local address 127.20.116.188:41705)
0928 14:53:42.986372 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.986374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.986385 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:53:42.986408 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.986409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.986440 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.986513 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:42.986891 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.987560 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.988691 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.989109 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.989193 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.989269 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:42.989313 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 14:53:42.989347 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:53:42.989408 27646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.986109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:41705 (local address 127.0.0.1:42374)
0928 14:53:42.986193 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.986204 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.986238 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.986527 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.986560 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.986562 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:42.986603 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.986605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.987630 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.987632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.988613 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.988615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.989192 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.989193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.989248 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:42.989251 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:53:42.989305 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 14:53:42.989346 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 14:53:42.989727 27888 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 14:53:42.989881 27918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2947c96f35da45d4b4bb59aeda6a1e0d. Latest consensus state: current_term: 1 leader_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: NON_VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: true } } }
I20260928 14:53:42.989979 27918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [sys.catalog]: This master's current role is: LEARNER
I20260928 14:53:42.988464 27086 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:53:42.990314 27086 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:53:42.991070 27843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: NON_VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: true } } }
I20260928 14:53:42.991176 27843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:42.992455 27843 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d: attempting to promote NON_VOTER 62ebad5801fc4460bfc70a394730e0b1 to VOTER
I20260928 14:53:42.992602 27843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: false } }
I20260928 14:53:42.993034 27696 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 1 FOLLOWER]: Refusing update from remote peer 2947c96f35da45d4b4bb59aeda6a1e0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 14:53:42.993257 27647 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 14:53:42.993379 27888 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 1 LEARNER]: Refusing update from remote peer 2947c96f35da45d4b4bb59aeda6a1e0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 14:53:42.993616 27647 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 14:53:42.993844 27648 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 62ebad5801fc4460bfc70a394730e0b1 (127.20.116.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: false } } }
I20260928 14:53:42.994107 27696 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 62ebad5801fc4460bfc70a394730e0b1 (127.20.116.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: false } } }
I20260928 14:53:42.994864 27648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: false } } }
I20260928 14:53:42.994956 27648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:42.994910 27843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: false } } }
I20260928 14:53:42.995018 27843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:42.995157 27842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: false } } }
I20260928 14:53:42.995253 27842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:53:42.995795 27647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: false } } }
I20260928 14:53:42.995883 27647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:42.996538 27888 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 62ebad5801fc4460bfc70a394730e0b1 (127.20.116.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: false } } }
I20260928 14:53:42.997697 27918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: false } } }
I20260928 14:53:42.997895 27918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:53:43.001082 27646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.996451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:35257 (local address 127.0.0.1:50298)
0928 14:53:42.996488 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.996513 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.996553 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.997185 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.997186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.997188 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:42.997245 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.997246 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.998658 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.998659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:42.999863 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.999866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.000814 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.000815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:43.000881 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:43.000885 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:53:43.000965 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 14:53:43.001004 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 14:53:43.001111 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.996619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50298 (local address 127.20.116.129:35257)
0928 14:53:42.997007 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.997011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.997021 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:53:42.997063 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.997065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.997082 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.997321 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:42.997391 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.998459 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.999967 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.000728 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.000827 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:43.000893 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:43.000982 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 14:53:43.001022 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":115,"threads_started":1}
I20260928 14:53:43.002036 27992 tablet_service.cc:1467] Processing CreateTablet for tablet e2804da8171d4b5cbd3f2ed144d56c86 (DEFAULT_TABLE table=test-table [id=3d4b2ea475a346249fb794db05a124d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:43.002501 28358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.996451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:42409 (local address 127.0.0.1:50776)
0928 14:53:42.998643 (+ 2192us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.998654 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.998739 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.998996 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.998998 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.999000 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:42.999048 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.999050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.999960 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:42.999960 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:43.001078 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:43.001081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.002240 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.002240 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:43.002328 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:43.002333 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:53:43.002382 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 14:53:43.002425 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2167,"mutex_wait_us":391,"thread_start_us":69,"threads_started":1}
I20260928 14:53:43.002511 27992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2804da8171d4b5cbd3f2ed144d56c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:43.002555 28356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.996640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.20.116.131:42409)
0928 14:53:42.996916 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.996922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.998739 (+ 1817us) server_negotiation.cc:407] Connection header received
0928 14:53:42.998819 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.998822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.998841 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.998917 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:42.999139 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:42.999841 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.001180 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.001902 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.002008 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:43.002062 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:43.002402 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 14:53:43.002448 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":96,"threads_started":1}
I20260928 14:53:43.002749 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.997013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:35679 (local address 127.0.0.1:53896)
0928 14:53:42.998740 (+ 1727us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:42.998751 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:42.998817 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:42.999116 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:42.999117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:42.999119 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:42.999167 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:42.999169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.000292 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.000293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:43.001440 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:43.001443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.002456 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.002457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:43.002520 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:43.002566 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 14:53:43.002633 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 14:53:43.002680 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1706,"thread_start_us":68,"threads_started":1}
I20260928 14:53:43.002861 28359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:42.997204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.20.116.130:35679)
0928 14:53:42.997381 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:42.997386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:42.998887 (+ 1501us) server_negotiation.cc:407] Connection header received
0928 14:53:42.998939 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:42.998942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:42.998959 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:42.999034 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:42.999245 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.000207 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.001507 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.002199 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.002363 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:43.002427 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:43.002698 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 14:53:43.002759 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":112,"threads_started":1}
I20260928 14:53:43.005626 28360 tablet_bootstrap.cc:492] T e2804da8171d4b5cbd3f2ed144d56c86 P dd1d5fa791f449278948c4abbb832e43: Bootstrap starting.
I20260928 14:53:43.006314 28360 tablet_bootstrap.cc:654] T e2804da8171d4b5cbd3f2ed144d56c86 P dd1d5fa791f449278948c4abbb832e43: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:43.006408 28129 tablet_service.cc:1467] Processing CreateTablet for tablet e2804da8171d4b5cbd3f2ed144d56c86 (DEFAULT_TABLE table=test-table [id=3d4b2ea475a346249fb794db05a124d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:43.006426 28265 tablet_service.cc:1467] Processing CreateTablet for tablet e2804da8171d4b5cbd3f2ed144d56c86 (DEFAULT_TABLE table=test-table [id=3d4b2ea475a346249fb794db05a124d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:43.006752 28360 log.cc:824] T e2804da8171d4b5cbd3f2ed144d56c86 P dd1d5fa791f449278948c4abbb832e43: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:43.006819 28265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2804da8171d4b5cbd3f2ed144d56c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:43.006819 28129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2804da8171d4b5cbd3f2ed144d56c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:43.007562 28360 tablet_bootstrap.cc:492] T e2804da8171d4b5cbd3f2ed144d56c86 P dd1d5fa791f449278948c4abbb832e43: No bootstrap required, opened a new log
I20260928 14:53:43.007651 28360 ts_tablet_manager.cc:1397] T e2804da8171d4b5cbd3f2ed144d56c86 P dd1d5fa791f449278948c4abbb832e43: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 14:53:43.009194 28360 raft_consensus.cc:348] T e2804da8171d4b5cbd3f2ed144d56c86 P dd1d5fa791f449278948c4abbb832e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd1d5fa791f449278948c4abbb832e43" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35257 } } peers { permanent_uuid: "c0148ddeb6e042fbb5ad121e91de2860" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 42409 } } peers { permanent_uuid: "d18126ddd7c840049afba377de0fa029" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35679 } }
I20260928 14:53:43.009358 28360 raft_consensus.cc:374] T e2804da8171d4b5cbd3f2ed144d56c86 P dd1d5fa791f449278948c4abbb832e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:43.009402 28360 raft_consensus.cc:729] T e2804da8171d4b5cbd3f2ed144d56c86 P dd1d5fa791f449278948c4abbb832e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd1d5fa791f449278948c4abbb832e43, State: Initialized, Role: FOLLOWER
I20260928 14:53:43.009538 28360 consensus_queue.cc:260] T e2804da8171d4b5cbd3f2ed144d56c86 P dd1d5fa791f449278948c4abbb832e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd1d5fa791f449278948c4abbb832e43" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35257 } } peers { permanent_uuid: "c0148ddeb6e042fbb5ad121e91de2860" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 42409 } } peers { permanent_uuid: "d18126ddd7c840049afba377de0fa029" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35679 } }
I20260928 14:53:43.010046 28059 heartbeater.cc:503] Master 127.20.116.190:35947 was elected leader, sending a full tablet report...
I20260928 14:53:43.011063 28360 ts_tablet_manager.cc:1428] T e2804da8171d4b5cbd3f2ed144d56c86 P dd1d5fa791f449278948c4abbb832e43: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 14:53:43.011296 28363 tablet_bootstrap.cc:492] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860: Bootstrap starting.
I20260928 14:53:43.012117 28363 tablet_bootstrap.cc:654] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:43.012427 28363 log.cc:824] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:43.012802 28364 tablet_bootstrap.cc:492] T e2804da8171d4b5cbd3f2ed144d56c86 P d18126ddd7c840049afba377de0fa029: Bootstrap starting.
I20260928 14:53:43.013211 28363 tablet_bootstrap.cc:492] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860: No bootstrap required, opened a new log
I20260928 14:53:43.013266 28363 ts_tablet_manager.cc:1397] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:53:43.013362 28364 tablet_bootstrap.cc:654] T e2804da8171d4b5cbd3f2ed144d56c86 P d18126ddd7c840049afba377de0fa029: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:43.013638 28364 log.cc:824] T e2804da8171d4b5cbd3f2ed144d56c86 P d18126ddd7c840049afba377de0fa029: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:43.014317 28364 tablet_bootstrap.cc:492] T e2804da8171d4b5cbd3f2ed144d56c86 P d18126ddd7c840049afba377de0fa029: No bootstrap required, opened a new log
I20260928 14:53:43.014394 28364 ts_tablet_manager.cc:1397] T e2804da8171d4b5cbd3f2ed144d56c86 P d18126ddd7c840049afba377de0fa029: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:53:43.015247 28363 raft_consensus.cc:348] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd1d5fa791f449278948c4abbb832e43" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35257 } } peers { permanent_uuid: "c0148ddeb6e042fbb5ad121e91de2860" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 42409 } } peers { permanent_uuid: "d18126ddd7c840049afba377de0fa029" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35679 } }
I20260928 14:53:43.015372 28363 raft_consensus.cc:374] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:43.015393 28363 raft_consensus.cc:729] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0148ddeb6e042fbb5ad121e91de2860, State: Initialized, Role: FOLLOWER
I20260928 14:53:43.015512 28363 consensus_queue.cc:260] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd1d5fa791f449278948c4abbb832e43" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35257 } } peers { permanent_uuid: "c0148ddeb6e042fbb5ad121e91de2860" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 42409 } } peers { permanent_uuid: "d18126ddd7c840049afba377de0fa029" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35679 } }
I20260928 14:53:43.015785 28363 ts_tablet_manager.cc:1428] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 14:53:43.015841 28332 heartbeater.cc:503] Master 127.20.116.190:35947 was elected leader, sending a full tablet report...
I20260928 14:53:43.017318 28364 raft_consensus.cc:348] T e2804da8171d4b5cbd3f2ed144d56c86 P d18126ddd7c840049afba377de0fa029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd1d5fa791f449278948c4abbb832e43" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35257 } } peers { permanent_uuid: "c0148ddeb6e042fbb5ad121e91de2860" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 42409 } } peers { permanent_uuid: "d18126ddd7c840049afba377de0fa029" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35679 } }
I20260928 14:53:43.017489 28364 raft_consensus.cc:374] T e2804da8171d4b5cbd3f2ed144d56c86 P d18126ddd7c840049afba377de0fa029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:43.017529 28364 raft_consensus.cc:729] T e2804da8171d4b5cbd3f2ed144d56c86 P d18126ddd7c840049afba377de0fa029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d18126ddd7c840049afba377de0fa029, State: Initialized, Role: FOLLOWER
I20260928 14:53:43.017666 28364 consensus_queue.cc:260] T e2804da8171d4b5cbd3f2ed144d56c86 P d18126ddd7c840049afba377de0fa029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd1d5fa791f449278948c4abbb832e43" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35257 } } peers { permanent_uuid: "c0148ddeb6e042fbb5ad121e91de2860" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 42409 } } peers { permanent_uuid: "d18126ddd7c840049afba377de0fa029" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35679 } }
I20260928 14:53:43.017947 28364 ts_tablet_manager.cc:1428] T e2804da8171d4b5cbd3f2ed144d56c86 P d18126ddd7c840049afba377de0fa029: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 14:53:43.017982 28196 heartbeater.cc:503] Master 127.20.116.190:35947 was elected leader, sending a full tablet report...
W20260928 14:53:43.071867 28198 tablet.cc:2369] T e2804da8171d4b5cbd3f2ed144d56c86 P d18126ddd7c840049afba377de0fa029: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:53:43.184242 28061 tablet.cc:2369] T e2804da8171d4b5cbd3f2ed144d56c86 P dd1d5fa791f449278948c4abbb832e43: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:53:43.218011 28334 tablet.cc:2369] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:53:43.258935 28367 raft_consensus.cc:482] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:53:43.259074 28367 raft_consensus.cc:504] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd1d5fa791f449278948c4abbb832e43" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35257 } } peers { permanent_uuid: "c0148ddeb6e042fbb5ad121e91de2860" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 42409 } } peers { permanent_uuid: "d18126ddd7c840049afba377de0fa029" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35679 } }
I20260928 14:53:43.259502 28367 leader_election.cc:288] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd1d5fa791f449278948c4abbb832e43 (127.20.116.129:35257), d18126ddd7c840049afba377de0fa029 (127.20.116.130:35679)
I20260928 14:53:43.262637 28359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:43.259803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:42869 (local address 127.20.116.130:35679)
0928 14:53:43.259870 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:43.259872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:43.260052 (+ 180us) server_negotiation.cc:407] Connection header received
0928 14:53:43.260115 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:43.260116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:43.260127 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:43.260186 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:43.260379 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.260879 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.261848 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.262235 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.262291 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:43.262299 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:43.262486 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 14:53:43.262560 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:53:43.262655 28336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:43.259737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:35679 (local address 127.20.116.131:42869)
0928 14:53:43.259979 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:43.259990 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:43.260046 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:43.260220 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:43.260221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:43.260229 (+ 8us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:43.260290 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:43.260292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.260985 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.260986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:43.261754 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:43.261757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.262325 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.262326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:43.262408 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:43.262411 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:53:43.262461 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 14:53:43.262493 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":188}
I20260928 14:53:43.262723 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:43.259799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:42613 (local address 127.20.116.129:35257)
0928 14:53:43.259873 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:43.259875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:43.259884 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:53:43.259911 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:43.259912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:43.259920 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:43.260011 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:43.260223 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.260772 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.261908 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.262261 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.262358 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:43.262367 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:43.262583 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 14:53:43.262637 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:53:43.262742 28337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:43.259737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:35257 (local address 127.20.116.131:42613)
0928 14:53:43.259789 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:43.259801 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:43.259854 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:43.260059 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:43.260060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:43.260062 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:43.260143 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:43.260146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.260860 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.260861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:43.261765 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:43.261768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.262374 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.262375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:43.262442 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:43.262465 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 14:53:43.262524 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 14:53:43.262585 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":47}
I20260928 14:53:43.263073 28149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2804da8171d4b5cbd3f2ed144d56c86" candidate_uuid: "c0148ddeb6e042fbb5ad121e91de2860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d18126ddd7c840049afba377de0fa029" is_pre_election: true
I20260928 14:53:43.263072 28012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2804da8171d4b5cbd3f2ed144d56c86" candidate_uuid: "c0148ddeb6e042fbb5ad121e91de2860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd1d5fa791f449278948c4abbb832e43" is_pre_election: true
I20260928 14:53:43.263250 28012 raft_consensus.cc:2454] T e2804da8171d4b5cbd3f2ed144d56c86 P dd1d5fa791f449278948c4abbb832e43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0148ddeb6e042fbb5ad121e91de2860 in term 0.
I20260928 14:53:43.263252 28149 raft_consensus.cc:2454] T e2804da8171d4b5cbd3f2ed144d56c86 P d18126ddd7c840049afba377de0fa029 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0148ddeb6e042fbb5ad121e91de2860 in term 0.
I20260928 14:53:43.263486 28221 leader_election.cc:302] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0148ddeb6e042fbb5ad121e91de2860, dd1d5fa791f449278948c4abbb832e43; no voters:
I20260928 14:53:43.263618 28367 raft_consensus.cc:2781] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:53:43.263725 28367 raft_consensus.cc:482] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:53:43.263765 28367 raft_consensus.cc:3037] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:43.264498 28367 raft_consensus.cc:504] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd1d5fa791f449278948c4abbb832e43" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35257 } } peers { permanent_uuid: "c0148ddeb6e042fbb5ad121e91de2860" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 42409 } } peers { permanent_uuid: "d18126ddd7c840049afba377de0fa029" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35679 } }
I20260928 14:53:43.264660 28367 leader_election.cc:288] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860 [CANDIDATE]: Term 1 election: Requested vote from peers dd1d5fa791f449278948c4abbb832e43 (127.20.116.129:35257), d18126ddd7c840049afba377de0fa029 (127.20.116.130:35679)
I20260928 14:53:43.264817 28149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2804da8171d4b5cbd3f2ed144d56c86" candidate_uuid: "c0148ddeb6e042fbb5ad121e91de2860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d18126ddd7c840049afba377de0fa029"
I20260928 14:53:43.264820 28012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2804da8171d4b5cbd3f2ed144d56c86" candidate_uuid: "c0148ddeb6e042fbb5ad121e91de2860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd1d5fa791f449278948c4abbb832e43"
I20260928 14:53:43.264905 28149 raft_consensus.cc:3037] T e2804da8171d4b5cbd3f2ed144d56c86 P d18126ddd7c840049afba377de0fa029 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:43.264928 28012 raft_consensus.cc:3037] T e2804da8171d4b5cbd3f2ed144d56c86 P dd1d5fa791f449278948c4abbb832e43 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:43.265626 28149 raft_consensus.cc:2454] T e2804da8171d4b5cbd3f2ed144d56c86 P d18126ddd7c840049afba377de0fa029 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0148ddeb6e042fbb5ad121e91de2860 in term 1.
I20260928 14:53:43.265691 28012 raft_consensus.cc:2454] T e2804da8171d4b5cbd3f2ed144d56c86 P dd1d5fa791f449278948c4abbb832e43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0148ddeb6e042fbb5ad121e91de2860 in term 1.
I20260928 14:53:43.265856 28221 leader_election.cc:302] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0148ddeb6e042fbb5ad121e91de2860, dd1d5fa791f449278948c4abbb832e43; no voters:
I20260928 14:53:43.266003 28367 raft_consensus.cc:2781] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:43.266171 28367 raft_consensus.cc:686] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860 [term 1 LEADER]: Becoming Leader. State: Replica: c0148ddeb6e042fbb5ad121e91de2860, State: Running, Role: LEADER
I20260928 14:53:43.266398 28367 consensus_queue.cc:237] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd1d5fa791f449278948c4abbb832e43" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35257 } } peers { permanent_uuid: "c0148ddeb6e042fbb5ad121e91de2860" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 42409 } } peers { permanent_uuid: "d18126ddd7c840049afba377de0fa029" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35679 } }
I20260928 14:53:43.267437 27086 catalog_manager.cc:5495] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0148ddeb6e042fbb5ad121e91de2860 (127.20.116.131). New cstate: current_term: 1 leader_uuid: "c0148ddeb6e042fbb5ad121e91de2860" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd1d5fa791f449278948c4abbb832e43" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0148ddeb6e042fbb5ad121e91de2860" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 42409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d18126ddd7c840049afba377de0fa029" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35679 } health_report { overall_health: UNKNOWN } } }
I20260928 14:53:43.284981 28356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:43.282336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50780 (local address 127.20.116.131:42409)
0928 14:53:43.282429 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:43.282431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:43.282442 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:53:43.282483 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:43.282484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:43.282512 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:43.282648 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:43.282830 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.283335 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.284082 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.284281 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.284339 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:43.284514 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:43.284643 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:43.284723 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:43.284815 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 14:53:43.284837 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:53:43.286927 28149 raft_consensus.cc:1261] T e2804da8171d4b5cbd3f2ed144d56c86 P d18126ddd7c840049afba377de0fa029 [term 1 FOLLOWER]: Refusing update from remote peer c0148ddeb6e042fbb5ad121e91de2860: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:43.286966 28012 raft_consensus.cc:1261] T e2804da8171d4b5cbd3f2ed144d56c86 P dd1d5fa791f449278948c4abbb832e43 [term 1 FOLLOWER]: Refusing update from remote peer c0148ddeb6e042fbb5ad121e91de2860: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:43.287322 28369 consensus_queue.cc:1035] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd1d5fa791f449278948c4abbb832e43" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:43.287463 28367 consensus_queue.cc:1035] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d18126ddd7c840049afba377de0fa029" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:43.293433 27086 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:43.293529 27086 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:43.295006 27086 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2947c96f35da45d4b4bb59aeda6a1e0d: Sending DeleteTablet for 3 replicas of tablet e2804da8171d4b5cbd3f2ed144d56c86
I20260928 14:53:43.295375 28129 tablet_service.cc:1514] Processing DeleteTablet for tablet e2804da8171d4b5cbd3f2ed144d56c86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:43 UTC) from {username='oryx', principal='oryx/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:53896
I20260928 14:53:43.295367 27992 tablet_service.cc:1514] Processing DeleteTablet for tablet e2804da8171d4b5cbd3f2ed144d56c86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:43 UTC) from {username='oryx', principal='oryx/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:50298
I20260928 14:53:43.295485 28265 tablet_service.cc:1514] Processing DeleteTablet for tablet e2804da8171d4b5cbd3f2ed144d56c86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:43 UTC) from {username='oryx', principal='oryx/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:50776
I20260928 14:53:43.295759 28382 tablet_replica.cc:330] T e2804da8171d4b5cbd3f2ed144d56c86 P dd1d5fa791f449278948c4abbb832e43: stopping tablet replica
I20260928 14:53:43.295854 28382 raft_consensus.cc:2229] T e2804da8171d4b5cbd3f2ed144d56c86 P dd1d5fa791f449278948c4abbb832e43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:43.295928 28382 raft_consensus.cc:2258] T e2804da8171d4b5cbd3f2ed144d56c86 P dd1d5fa791f449278948c4abbb832e43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:43.296533 28382 ts_tablet_manager.cc:1905] T e2804da8171d4b5cbd3f2ed144d56c86 P dd1d5fa791f449278948c4abbb832e43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:43.296942 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 27054
I20260928 14:53:43.298503 28382 ts_tablet_manager.cc:1918] T e2804da8171d4b5cbd3f2ed144d56c86 P dd1d5fa791f449278948c4abbb832e43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:53:43.298596 28382 log.cc:1196] T e2804da8171d4b5cbd3f2ed144d56c86 P dd1d5fa791f449278948c4abbb832e43: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-0/wal/wals/e2804da8171d4b5cbd3f2ed144d56c86
I20260928 14:53:43.298852 28382 ts_tablet_manager.cc:1939] T e2804da8171d4b5cbd3f2ed144d56c86 P dd1d5fa791f449278948c4abbb832e43: Deleting consensus metadata
I20260928 14:53:43.302711 28384 tablet_replica.cc:330] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860: stopping tablet replica
I20260928 14:53:43.302857 28384 raft_consensus.cc:2229] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:53:43.302984 28384 raft_consensus.cc:2258] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:43.303227 28383 tablet_replica.cc:330] T e2804da8171d4b5cbd3f2ed144d56c86 P d18126ddd7c840049afba377de0fa029: stopping tablet replica
I20260928 14:53:43.303350 28383 raft_consensus.cc:2229] T e2804da8171d4b5cbd3f2ed144d56c86 P d18126ddd7c840049afba377de0fa029 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:43.303481 28383 raft_consensus.cc:2258] T e2804da8171d4b5cbd3f2ed144d56c86 P d18126ddd7c840049afba377de0fa029 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:43.304325 28065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:43.296103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42378 (local address 127.20.116.188:41705)
0928 14:53:43.296144 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:43.296147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:43.296204 (+ 57us) server_negotiation.cc:407] Connection header received
0928 14:53:43.296306 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:43.296307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:43.296318 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:43.296348 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:43.297301 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.298035 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.299202 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.299463 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.299510 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:43.301189 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:43.301190 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:43.301191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:43.301198 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:43.302872 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:43.303498 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:43.303500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:43.303501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:43.303563 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:43.303784 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:43.303785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:43.303786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:43.303912 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:43.304029 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:43.304133 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 14:53:43.304219 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 14:53:43.306216 27945 connection.cc:535] server connection from 127.0.0.1:50298 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:53:43.308334 28384 ts_tablet_manager.cc:1905] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:43.310195 28384 ts_tablet_manager.cc:1918] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:53:43.310276 28384 log.cc:1196] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/e2804da8171d4b5cbd3f2ed144d56c86
I20260928 14:53:43.310598 28384 ts_tablet_manager.cc:1939] T e2804da8171d4b5cbd3f2ed144d56c86 P c0148ddeb6e042fbb5ad121e91de2860: Deleting consensus metadata
W20260928 14:53:43.310972 28219 connection.cc:441] server connection from 127.0.0.1:50776 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50776 (request call id 1) could send its response
I20260928 14:53:43.311317 28383 ts_tablet_manager.cc:1905] T e2804da8171d4b5cbd3f2ed144d56c86 P d18126ddd7c840049afba377de0fa029: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:43.313027 28383 ts_tablet_manager.cc:1918] T e2804da8171d4b5cbd3f2ed144d56c86 P d18126ddd7c840049afba377de0fa029: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:53:43.313112 28383 log.cc:1196] T e2804da8171d4b5cbd3f2ed144d56c86 P d18126ddd7c840049afba377de0fa029: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/e2804da8171d4b5cbd3f2ed144d56c86
I20260928 14:53:43.313385 28383 ts_tablet_manager.cc:1939] T e2804da8171d4b5cbd3f2ed144d56c86 P d18126ddd7c840049afba377de0fa029: Deleting consensus metadata
W20260928 14:53:43.313766 28082 connection.cc:441] server connection from 127.0.0.1:53896 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53896 (request call id 1) could send its response
I20260928 14:53:43.316939 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:43.308131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33238 (local address 127.20.116.189:37691)
0928 14:53:43.308178 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:43.308180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:43.308343 (+ 163us) server_negotiation.cc:407] Connection header received
0928 14:53:43.308623 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:43.308624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:43.308643 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:43.308699 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:43.309565 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.310334 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.311547 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.311863 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.311904 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:43.313719 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:43.313721 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:43.313721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:43.313729 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:43.315624 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:43.315993 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:43.315995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:43.315995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:43.316063 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:43.316271 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:43.316272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:43.316272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:43.316413 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:43.316523 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:43.316740 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 14:53:43.316840 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:53:43.317190 28065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:43.308461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42380 (local address 127.20.116.188:41705)
0928 14:53:43.308499 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:43.308500 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:43.308688 (+ 188us) server_negotiation.cc:407] Connection header received
0928 14:53:43.308782 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:43.308783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:43.308793 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:43.308823 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:43.309561 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.310333 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.311656 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:43.311941 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:43.311979 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:43.313805 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:43.313806 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:43.313806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:43.313813 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:43.315906 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:43.316329 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:43.316331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:43.316331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:43.316383 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:43.316612 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:43.316612 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:43.316613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:43.316748 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:43.316833 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:43.317015 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 14:53:43.317109 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:53:44.292570 28392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:44.292119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.129:42681)
0928 14:53:44.292384 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:44.292516 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":108,"threads_started":1}
I20260928 14:53:44.292570 28393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:44.292330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.130:53861)
0928 14:53:44.292435 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:44.292513 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":48,"threads_started":1}
W20260928 14:53:44.292943 28196 heartbeater.cc:644] Failed to heartbeat to 127.20.116.190:35947 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
W20260928 14:53:44.292929 28059 heartbeater.cc:644] Failed to heartbeat to 127.20.116.190:35947 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
I20260928 14:53:44.293213 28394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:44.292947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.131:37675)
0928 14:53:44.293083 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:44.293163 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
W20260928 14:53:44.293406 28332 heartbeater.cc:644] Failed to heartbeat to 127.20.116.190:35947 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
I20260928 14:53:44.795758 28395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2947c96f35da45d4b4bb59aeda6a1e0d)
I20260928 14:53:44.795961 28395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: false } }
I20260928 14:53:44.795936 28396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2947c96f35da45d4b4bb59aeda6a1e0d)
I20260928 14:53:44.796139 28396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: false } }
I20260928 14:53:44.796330 28395 leader_election.cc:288] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2947c96f35da45d4b4bb59aeda6a1e0d (127.20.116.190:35947), 62ebad5801fc4460bfc70a394730e0b1 (127.20.116.188:41705)
I20260928 14:53:44.796730 28398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:44.796480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.0.0.1:33370)
0928 14:53:44.796614 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:44.796684 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
W20260928 14:53:44.796893 27673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
I20260928 14:53:44.797086 28396 leader_election.cc:288] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2947c96f35da45d4b4bb59aeda6a1e0d (127.20.116.190:35947), 51131b65b43344d59beff206ed38e6bd (127.20.116.189:37691)
I20260928 14:53:44.797391 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:44.796481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.0.0.1:33374)
0928 14:53:44.796742 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:44.797226 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":131,"threads_started":1}
I20260928 14:53:44.797451 28398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:44.797334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.0.0.1:33388)
0928 14:53:44.797393 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:44.797426 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 14:53:44.797569 27673 leader_election.cc:334] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2947c96f35da45d4b4bb59aeda6a1e0d (127.20.116.190:35947): Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
W20260928 14:53:44.797725 27866 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
I20260928 14:53:44.798166 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:44.798053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.0.0.1:33390)
0928 14:53:44.798110 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:44.798137 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 14:53:44.799099 27866 leader_election.cc:334] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2947c96f35da45d4b4bb59aeda6a1e0d (127.20.116.190:35947): Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
I20260928 14:53:44.800045 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:44.796651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42392 (local address 127.20.116.188:41705)
0928 14:53:44.796818 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:44.796820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:44.796829 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:53:44.796858 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:44.796859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:44.796870 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:44.796936 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:44.797161 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:44.797909 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:44.799081 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:44.799643 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:44.799719 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:44.799728 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:44.799926 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 14:53:44.799971 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":43,"threads_started":1}
I20260928 14:53:44.800077 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:44.796484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:41705 (local address 127.0.0.1:42392)
0928 14:53:44.796643 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:44.796655 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:44.796722 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:44.796994 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:44.796995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:44.796998 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:44.797059 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:44.797061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:44.798019 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:44.798020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:44.798997 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:44.798999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:44.799736 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:44.799737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:44.799838 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:44.799844 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:53:44.799913 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 14:53:44.799951 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":61,"threads_started":1}
I20260928 14:53:44.800129 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:44.796836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:37691 (local address 127.0.0.1:33252)
0928 14:53:44.796934 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:44.796944 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:44.797000 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:44.797254 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:44.797255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:44.797259 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:44.797317 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:44.797318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:44.798234 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:44.798235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:44.799102 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:44.799104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:44.799879 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:44.799880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:44.799962 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:44.799965 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:53:44.800020 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 14:53:44.800061 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":65,"threads_started":1}
I20260928 14:53:44.800364 28403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:44.796979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33252 (local address 127.20.116.189:37691)
0928 14:53:44.797114 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:44.797116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:44.797123 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:53:44.797144 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:44.797145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:44.797155 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:44.797395 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:44.797429 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:44.798148 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:44.799218 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:44.799782 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:44.799837 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:44.799843 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:44.800039 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 14:53:44.800076 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260928 14:53:44.800323 27888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51131b65b43344d59beff206ed38e6bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "62ebad5801fc4460bfc70a394730e0b1" is_pre_election: true
I20260928 14:53:44.800438 27888 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51131b65b43344d59beff206ed38e6bd in term 1.
I20260928 14:53:44.800573 27671 leader_election.cc:302] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51131b65b43344d59beff206ed38e6bd, 62ebad5801fc4460bfc70a394730e0b1; no voters: 2947c96f35da45d4b4bb59aeda6a1e0d
I20260928 14:53:44.800516 27696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ebad5801fc4460bfc70a394730e0b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "51131b65b43344d59beff206ed38e6bd" is_pre_election: true
I20260928 14:53:44.800704 27696 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62ebad5801fc4460bfc70a394730e0b1 in term 1.
I20260928 14:53:44.800779 28395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 14:53:44.800830 28395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2947c96f35da45d4b4bb59aeda6a1e0d)
I20260928 14:53:44.800863 28395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:53:44.800901 27865 leader_election.cc:302] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51131b65b43344d59beff206ed38e6bd, 62ebad5801fc4460bfc70a394730e0b1; no voters: 2947c96f35da45d4b4bb59aeda6a1e0d
I20260928 14:53:44.801007 28396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 14:53:44.801043 28396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2947c96f35da45d4b4bb59aeda6a1e0d)
I20260928 14:53:44.801064 28396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:53:44.801717 28395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: false } }
I20260928 14:53:44.801877 28395 leader_election.cc:288] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [CANDIDATE]: Term 2 election: Requested vote from peers 2947c96f35da45d4b4bb59aeda6a1e0d (127.20.116.190:35947), 62ebad5801fc4460bfc70a394730e0b1 (127.20.116.188:41705)
I20260928 14:53:44.801920 28396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: false } }
I20260928 14:53:44.802081 28396 leader_election.cc:288] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [CANDIDATE]: Term 2 election: Requested vote from peers 2947c96f35da45d4b4bb59aeda6a1e0d (127.20.116.190:35947), 51131b65b43344d59beff206ed38e6bd (127.20.116.189:37691)
I20260928 14:53:44.802127 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:44.802029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.0.0.1:33412)
0928 14:53:44.802065 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:44.802095 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 14:53:44.802237 27696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ebad5801fc4460bfc70a394730e0b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "51131b65b43344d59beff206ed38e6bd"
I20260928 14:53:44.802325 27696 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 62ebad5801fc4460bfc70a394730e0b1 in current term 2: Already voted for candidate 51131b65b43344d59beff206ed38e6bd in this term.
I20260928 14:53:44.802345 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:44.802214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.0.0.1:33424)
0928 14:53:44.802285 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:44.802315 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 14:53:44.802362 27888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51131b65b43344d59beff206ed38e6bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "62ebad5801fc4460bfc70a394730e0b1"
I20260928 14:53:44.802467 27888 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 51131b65b43344d59beff206ed38e6bd in current term 2: Already voted for candidate 62ebad5801fc4460bfc70a394730e0b1 in this term.
I20260928 14:53:44.802672 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:44.802583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.0.0.1:33430)
0928 14:53:44.802616 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:44.802643 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 14:53:44.802743 27866 leader_election.cc:334] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2947c96f35da45d4b4bb59aeda6a1e0d (127.20.116.190:35947): Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
I20260928 14:53:44.802766 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:44.802679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.0.0.1:33442)
0928 14:53:44.802710 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:44.802737 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 14:53:44.802793 27866 leader_election.cc:302] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ebad5801fc4460bfc70a394730e0b1; no voters: 2947c96f35da45d4b4bb59aeda6a1e0d, 51131b65b43344d59beff206ed38e6bd
I20260928 14:53:44.802863 28396 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260928 14:53:44.802906 27673 leader_election.cc:334] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2947c96f35da45d4b4bb59aeda6a1e0d (127.20.116.190:35947): Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
I20260928 14:53:44.802939 27673 leader_election.cc:302] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51131b65b43344d59beff206ed38e6bd; no voters: 2947c96f35da45d4b4bb59aeda6a1e0d, 62ebad5801fc4460bfc70a394730e0b1
I20260928 14:53:44.803002 28395 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 14:53:45.293519 28405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:45.293272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.130:56671)
0928 14:53:45.293404 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:45.293450 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260928 14:53:45.293546 28406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:45.293310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.129:55419)
0928 14:53:45.293441 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:45.293494 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260928 14:53:45.294078 28407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:45.293776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.131:38477)
0928 14:53:45.293999 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:45.294035 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":68,"threads_started":1}
I20260928 14:53:46.294267 28408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.294020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.130:54357)
0928 14:53:46.294161 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.294206 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260928 14:53:46.294384 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.294142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.129:37831)
0928 14:53:46.294283 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.294329 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260928 14:53:46.294680 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.294460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.131:43755)
0928 14:53:46.294604 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.294641 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":47,"threads_started":1}
I20260928 14:53:46.420288 28411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:53:46.420421 28411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: false } }
I20260928 14:53:46.420650 28411 leader_election.cc:288] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2947c96f35da45d4b4bb59aeda6a1e0d (127.20.116.190:35947), 51131b65b43344d59beff206ed38e6bd (127.20.116.189:37691)
I20260928 14:53:46.421021 28412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.420812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.0.0.1:33672)
0928 14:53:46.420932 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.420979 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
I20260928 14:53:46.420977 27696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ebad5801fc4460bfc70a394730e0b1" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "51131b65b43344d59beff206ed38e6bd" is_pre_election: true
I20260928 14:53:46.421092 27696 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62ebad5801fc4460bfc70a394730e0b1 in term 2.
I20260928 14:53:46.421276 27865 leader_election.cc:302] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51131b65b43344d59beff206ed38e6bd, 62ebad5801fc4460bfc70a394730e0b1; no voters:
I20260928 14:53:46.421375 28411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 14:53:46.421411 28411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:53:46.421442 28411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 2 FOLLOWER]: Advancing to term 3
I20260928 14:53:46.421602 28412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.421450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.0.0.1:33678)
0928 14:53:46.421489 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.421521 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 14:53:46.421694 27866 leader_election.cc:334] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2947c96f35da45d4b4bb59aeda6a1e0d (127.20.116.190:35947): Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
I20260928 14:53:46.422292 28411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: false } }
I20260928 14:53:46.422509 28411 leader_election.cc:288] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [CANDIDATE]: Term 3 election: Requested vote from peers 2947c96f35da45d4b4bb59aeda6a1e0d (127.20.116.190:35947), 51131b65b43344d59beff206ed38e6bd (127.20.116.189:37691)
I20260928 14:53:46.422646 28412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.422535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.0.0.1:33692)
0928 14:53:46.422597 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.422622 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 14:53:46.422606 27696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ebad5801fc4460bfc70a394730e0b1" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "51131b65b43344d59beff206ed38e6bd"
I20260928 14:53:46.422696 27696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 2 FOLLOWER]: Advancing to term 3
I20260928 14:53:46.422981 28412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.422872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.0.0.1:33698)
0928 14:53:46.422929 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.422958 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 14:53:46.423074 27866 leader_election.cc:334] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 2947c96f35da45d4b4bb59aeda6a1e0d (127.20.116.190:35947): Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
I20260928 14:53:46.423486 27696 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62ebad5801fc4460bfc70a394730e0b1 in term 3.
I20260928 14:53:46.423666 27865 leader_election.cc:302] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51131b65b43344d59beff206ed38e6bd, 62ebad5801fc4460bfc70a394730e0b1; no voters: 2947c96f35da45d4b4bb59aeda6a1e0d
I20260928 14:53:46.423769 28411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 14:53:46.423882 28411 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [term 3 LEADER]: Becoming Leader. State: Replica: 62ebad5801fc4460bfc70a394730e0b1, State: Running, Role: LEADER
I20260928 14:53:46.423978 28411 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: false } }
I20260928 14:53:46.424361 28414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ebad5801fc4460bfc70a394730e0b1. Latest consensus state: current_term: 3 leader_uuid: "62ebad5801fc4460bfc70a394730e0b1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: false } } }
I20260928 14:53:46.424460 28414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:46.424649 28416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:53:46.425073 28416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:53:46.425168 28416 catalog_manager.cc:1259] Loaded cluster ID: b4b97e5e908a4355897c5dbba5724d5a
I20260928 14:53:46.425200 28416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:53:46.425386 28416 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:53:46.425482 28416 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1: Loaded TSK: 0
I20260928 14:53:46.425627 28416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:53:46.483312 27878 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:53:46.485605 27878 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:53:46.486953 27696 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [term 3 FOLLOWER]: Refusing update from remote peer 62ebad5801fc4460bfc70a394730e0b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 14:53:46.487171 28412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.486835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.0.0.1:33722)
0928 14:53:46.487100 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.487137 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242}
I20260928 14:53:46.487201 28411 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 14:53:46.487648 28412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.487543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.0.0.1:33724)
0928 14:53:46.487594 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.487621 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 14:53:46.487727 27866 consensus_peers.cc:487] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 -> Peer 2947c96f35da45d4b4bb59aeda6a1e0d (127.20.116.190:35947): Couldn't send request to peer 2947c96f35da45d4b4bb59aeda6a1e0d. Status: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 14:53:46.487902 28418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ebad5801fc4460bfc70a394730e0b1. Latest consensus state: current_term: 3 leader_uuid: "62ebad5801fc4460bfc70a394730e0b1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: false } } }
I20260928 14:53:46.488025 28418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:53:46.488703 28414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "62ebad5801fc4460bfc70a394730e0b1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: false } } }
I20260928 14:53:46.488797 28414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:46.488904 28419 mvcc.cc:205] Tried to move back new op lower bound from 7334327199688171520 to 7334327199433375744. Current Snapshot: MvccSnapshot[applied={T|T < 7334327199688171520}]
I20260928 14:53:46.489023 28418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "62ebad5801fc4460bfc70a394730e0b1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2947c96f35da45d4b4bb59aeda6a1e0d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35947 } } peers { permanent_uuid: "51131b65b43344d59beff206ed38e6bd" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 37691 } attrs { promote: false } } peers { permanent_uuid: "62ebad5801fc4460bfc70a394730e0b1" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 41705 } attrs { promote: false } } }
I20260928 14:53:46.489110 28418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51131b65b43344d59beff206ed38e6bd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:53:46.494360 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.490872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:42409 (local address 127.0.0.1:50782)
0928 14:53:46.491052 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.491062 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:46.491147 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:46.491335 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:46.491336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:46.491337 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:46.491371 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:46.491372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:46.492291 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:46.492292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:46.493313 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:46.493316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:46.494099 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:46.494100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:46.494162 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:46.494166 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:53:46.494240 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 14:53:46.494279 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":41,"thread_start_us":86,"threads_started":1}
I20260928 14:53:46.494390 28426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.490943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50782 (local address 127.20.116.131:42409)
0928 14:53:46.491057 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:46.491059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:46.491135 (+ 76us) server_negotiation.cc:407] Connection header received
0928 14:53:46.491203 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:46.491210 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:46.491222 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:46.491289 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:46.491445 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:46.492195 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:46.493405 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:46.493987 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:46.494057 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:46.494066 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:46.494251 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 14:53:46.494304 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":56,"threads_started":1}
I20260928 14:53:46.494840 28265 tablet_service.cc:1467] Processing CreateTablet for tablet ac8f7073506248b8bb9501bc0cfcf918 (DEFAULT_TABLE table=test-table [id=52fa3b4ca61747f7a9604b0db5dfc3af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:46.494987 28265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac8f7073506248b8bb9501bc0cfcf918. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:46.495584 28412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.490778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:35679 (local address 127.0.0.1:53908)
0928 14:53:46.491016 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.491037 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:46.491104 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:46.491256 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:46.491257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:46.491259 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:46.491318 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:46.491320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:46.492287 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:46.492288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:46.493490 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:46.493493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:46.495320 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:46.495321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:46.495402 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:46.495407 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:53:46.495472 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 14:53:46.495515 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":24}
I20260928 14:53:46.495600 28423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.490845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.20.116.130:35679)
0928 14:53:46.490959 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:46.490961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:46.491087 (+ 126us) server_negotiation.cc:407] Connection header received
0928 14:53:46.491165 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:46.491166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:46.491179 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:46.491254 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:46.491408 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:46.492196 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:46.493567 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:46.494140 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:46.494205 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:46.494214 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:46.495481 (+ 1267us) server_negotiation.cc:299] Negotiation successful
0928 14:53:46.495529 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":54,"threads_started":1}
I20260928 14:53:46.495842 28129 tablet_service.cc:1467] Processing CreateTablet for tablet ac8f7073506248b8bb9501bc0cfcf918 (DEFAULT_TABLE table=test-table [id=52fa3b4ca61747f7a9604b0db5dfc3af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:46.495958 28129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac8f7073506248b8bb9501bc0cfcf918. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:46.496582 28428 tablet_bootstrap.cc:492] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860: Bootstrap starting.
I20260928 14:53:46.497009 28428 tablet_bootstrap.cc:654] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:46.497275 28429 tablet_bootstrap.cc:492] T ac8f7073506248b8bb9501bc0cfcf918 P d18126ddd7c840049afba377de0fa029: Bootstrap starting.
I20260928 14:53:46.497475 28425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.490882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:35257 (local address 127.0.0.1:50300)
0928 14:53:46.491542 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.491679 (+ 137us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:46.491736 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:46.491884 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:46.491885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:46.491887 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:46.491945 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:46.491947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:46.492841 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:46.492842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:46.494590 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:46.494593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:46.495367 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:46.495368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:46.495423 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:46.495427 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:53:46.497349 (+ 1922us) client_negotiation.cc:240] Negotiation successful
0928 14:53:46.497394 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":93,"threads_started":1}
I20260928 14:53:46.497642 28429 tablet_bootstrap.cc:654] T ac8f7073506248b8bb9501bc0cfcf918 P d18126ddd7c840049afba377de0fa029: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:46.497750 28428 tablet_bootstrap.cc:492] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860: No bootstrap required, opened a new log
I20260928 14:53:46.497812 28428 ts_tablet_manager.cc:1397] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:46.497815 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.491300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50300 (local address 127.20.116.129:35257)
0928 14:53:46.491405 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:46.491407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:46.491728 (+ 321us) server_negotiation.cc:407] Connection header received
0928 14:53:46.491802 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:46.491803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:46.491814 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:46.491876 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:46.492005 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:46.492749 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:46.494679 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:46.495245 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:46.495324 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:46.495332 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:46.497576 (+ 2244us) server_negotiation.cc:299] Negotiation successful
0928 14:53:46.497732 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":60,"threads_started":1}
I20260928 14:53:46.497946 28428 raft_consensus.cc:348] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18126ddd7c840049afba377de0fa029" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35679 } } peers { permanent_uuid: "c0148ddeb6e042fbb5ad121e91de2860" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 42409 } } peers { permanent_uuid: "dd1d5fa791f449278948c4abbb832e43" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35257 } }
I20260928 14:53:46.498023 28428 raft_consensus.cc:374] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:46.498010 27992 tablet_service.cc:1467] Processing CreateTablet for tablet ac8f7073506248b8bb9501bc0cfcf918 (DEFAULT_TABLE table=test-table [id=52fa3b4ca61747f7a9604b0db5dfc3af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:46.498060 28428 raft_consensus.cc:729] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0148ddeb6e042fbb5ad121e91de2860, State: Initialized, Role: FOLLOWER
I20260928 14:53:46.498117 27992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac8f7073506248b8bb9501bc0cfcf918. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:46.498128 28428 consensus_queue.cc:260] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18126ddd7c840049afba377de0fa029" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35679 } } peers { permanent_uuid: "c0148ddeb6e042fbb5ad121e91de2860" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 42409 } } peers { permanent_uuid: "dd1d5fa791f449278948c4abbb832e43" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35257 } }
I20260928 14:53:46.498274 28429 tablet_bootstrap.cc:492] T ac8f7073506248b8bb9501bc0cfcf918 P d18126ddd7c840049afba377de0fa029: No bootstrap required, opened a new log
I20260928 14:53:46.498322 28428 ts_tablet_manager.cc:1428] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:53:46.498344 28429 ts_tablet_manager.cc:1397] T ac8f7073506248b8bb9501bc0cfcf918 P d18126ddd7c840049afba377de0fa029: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:46.498595 28429 raft_consensus.cc:348] T ac8f7073506248b8bb9501bc0cfcf918 P d18126ddd7c840049afba377de0fa029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18126ddd7c840049afba377de0fa029" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35679 } } peers { permanent_uuid: "c0148ddeb6e042fbb5ad121e91de2860" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 42409 } } peers { permanent_uuid: "dd1d5fa791f449278948c4abbb832e43" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35257 } }
I20260928 14:53:46.498668 28429 raft_consensus.cc:374] T ac8f7073506248b8bb9501bc0cfcf918 P d18126ddd7c840049afba377de0fa029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:46.498693 28429 raft_consensus.cc:729] T ac8f7073506248b8bb9501bc0cfcf918 P d18126ddd7c840049afba377de0fa029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d18126ddd7c840049afba377de0fa029, State: Initialized, Role: FOLLOWER
I20260928 14:53:46.498762 28429 consensus_queue.cc:260] T ac8f7073506248b8bb9501bc0cfcf918 P d18126ddd7c840049afba377de0fa029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18126ddd7c840049afba377de0fa029" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35679 } } peers { permanent_uuid: "c0148ddeb6e042fbb5ad121e91de2860" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 42409 } } peers { permanent_uuid: "dd1d5fa791f449278948c4abbb832e43" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35257 } }
I20260928 14:53:46.498950 28429 ts_tablet_manager.cc:1428] T ac8f7073506248b8bb9501bc0cfcf918 P d18126ddd7c840049afba377de0fa029: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:46.499025 28331 heartbeater.cc:503] Master 127.20.116.188:41705 was elected leader, sending a full tablet report...
I20260928 14:53:46.499604 28195 heartbeater.cc:503] Master 127.20.116.188:41705 was elected leader, sending a full tablet report...
I20260928 14:53:46.499868 28408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.499700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.130:50997)
0928 14:53:46.499806 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.499835 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 14:53:46.499883 28434 tablet_bootstrap.cc:492] T ac8f7073506248b8bb9501bc0cfcf918 P dd1d5fa791f449278948c4abbb832e43: Bootstrap starting.
I20260928 14:53:46.499996 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.499827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.131:46137)
0928 14:53:46.499934 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.499969 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 14:53:46.500461 28408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.500356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.130:57745)
0928 14:53:46.500395 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.500420 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 14:53:46.500697 28434 tablet_bootstrap.cc:654] T ac8f7073506248b8bb9501bc0cfcf918 P dd1d5fa791f449278948c4abbb832e43: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:46.501395 28434 tablet_bootstrap.cc:492] T ac8f7073506248b8bb9501bc0cfcf918 P dd1d5fa791f449278948c4abbb832e43: No bootstrap required, opened a new log
I20260928 14:53:46.501454 28434 ts_tablet_manager.cc:1397] T ac8f7073506248b8bb9501bc0cfcf918 P dd1d5fa791f449278948c4abbb832e43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:53:46.501616 28434 raft_consensus.cc:348] T ac8f7073506248b8bb9501bc0cfcf918 P dd1d5fa791f449278948c4abbb832e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18126ddd7c840049afba377de0fa029" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35679 } } peers { permanent_uuid: "c0148ddeb6e042fbb5ad121e91de2860" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 42409 } } peers { permanent_uuid: "dd1d5fa791f449278948c4abbb832e43" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35257 } }
I20260928 14:53:46.501698 28434 raft_consensus.cc:374] T ac8f7073506248b8bb9501bc0cfcf918 P dd1d5fa791f449278948c4abbb832e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:46.501721 28434 raft_consensus.cc:729] T ac8f7073506248b8bb9501bc0cfcf918 P dd1d5fa791f449278948c4abbb832e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd1d5fa791f449278948c4abbb832e43, State: Initialized, Role: FOLLOWER
I20260928 14:53:46.501782 28434 consensus_queue.cc:260] T ac8f7073506248b8bb9501bc0cfcf918 P dd1d5fa791f449278948c4abbb832e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18126ddd7c840049afba377de0fa029" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35679 } } peers { permanent_uuid: "c0148ddeb6e042fbb5ad121e91de2860" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 42409 } } peers { permanent_uuid: "dd1d5fa791f449278948c4abbb832e43" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35257 } }
I20260928 14:53:46.501978 28434 ts_tablet_manager.cc:1428] T ac8f7073506248b8bb9501bc0cfcf918 P dd1d5fa791f449278948c4abbb832e43: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:53:46.502321 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.502221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.129:47433)
0928 14:53:46.502268 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.502295 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 14:53:46.502367 28058 heartbeater.cc:503] Master 127.20.116.188:41705 was elected leader, sending a full tablet report...
I20260928 14:53:46.502611 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.502503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.129:36517)
0928 14:53:46.502558 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.502587 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 14:53:46.505394 28432 raft_consensus.cc:482] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:53:46.505491 28432 raft_consensus.cc:504] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18126ddd7c840049afba377de0fa029" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35679 } } peers { permanent_uuid: "c0148ddeb6e042fbb5ad121e91de2860" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 42409 } } peers { permanent_uuid: "dd1d5fa791f449278948c4abbb832e43" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35257 } }
I20260928 14:53:46.505662 28432 leader_election.cc:288] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d18126ddd7c840049afba377de0fa029 (127.20.116.130:35679), dd1d5fa791f449278948c4abbb832e43 (127.20.116.129:35257)
I20260928 14:53:46.505805 28149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8f7073506248b8bb9501bc0cfcf918" candidate_uuid: "c0148ddeb6e042fbb5ad121e91de2860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d18126ddd7c840049afba377de0fa029" is_pre_election: true
I20260928 14:53:46.505883 28149 raft_consensus.cc:2454] T ac8f7073506248b8bb9501bc0cfcf918 P d18126ddd7c840049afba377de0fa029 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0148ddeb6e042fbb5ad121e91de2860 in term 0.
I20260928 14:53:46.505856 28012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8f7073506248b8bb9501bc0cfcf918" candidate_uuid: "c0148ddeb6e042fbb5ad121e91de2860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd1d5fa791f449278948c4abbb832e43" is_pre_election: true
I20260928 14:53:46.505939 28012 raft_consensus.cc:2454] T ac8f7073506248b8bb9501bc0cfcf918 P dd1d5fa791f449278948c4abbb832e43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0148ddeb6e042fbb5ad121e91de2860 in term 0.
I20260928 14:53:46.506038 28220 leader_election.cc:302] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0148ddeb6e042fbb5ad121e91de2860, d18126ddd7c840049afba377de0fa029; no voters:
I20260928 14:53:46.506130 28432 raft_consensus.cc:2781] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:53:46.506182 28432 raft_consensus.cc:482] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:53:46.506212 28432 raft_consensus.cc:3037] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:46.507436 28432 raft_consensus.cc:504] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18126ddd7c840049afba377de0fa029" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35679 } } peers { permanent_uuid: "c0148ddeb6e042fbb5ad121e91de2860" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 42409 } } peers { permanent_uuid: "dd1d5fa791f449278948c4abbb832e43" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35257 } }
I20260928 14:53:46.507685 28432 leader_election.cc:288] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860 [CANDIDATE]: Term 1 election: Requested vote from peers d18126ddd7c840049afba377de0fa029 (127.20.116.130:35679), dd1d5fa791f449278948c4abbb832e43 (127.20.116.129:35257)
I20260928 14:53:46.507764 28149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8f7073506248b8bb9501bc0cfcf918" candidate_uuid: "c0148ddeb6e042fbb5ad121e91de2860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d18126ddd7c840049afba377de0fa029"
I20260928 14:53:46.507862 28149 raft_consensus.cc:3037] T ac8f7073506248b8bb9501bc0cfcf918 P d18126ddd7c840049afba377de0fa029 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:46.507845 28012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8f7073506248b8bb9501bc0cfcf918" candidate_uuid: "c0148ddeb6e042fbb5ad121e91de2860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd1d5fa791f449278948c4abbb832e43"
I20260928 14:53:46.507925 28012 raft_consensus.cc:3037] T ac8f7073506248b8bb9501bc0cfcf918 P dd1d5fa791f449278948c4abbb832e43 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:46.508704 28149 raft_consensus.cc:2454] T ac8f7073506248b8bb9501bc0cfcf918 P d18126ddd7c840049afba377de0fa029 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0148ddeb6e042fbb5ad121e91de2860 in term 1.
I20260928 14:53:46.508823 28012 raft_consensus.cc:2454] T ac8f7073506248b8bb9501bc0cfcf918 P dd1d5fa791f449278948c4abbb832e43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0148ddeb6e042fbb5ad121e91de2860 in term 1.
I20260928 14:53:46.508898 28220 leader_election.cc:302] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0148ddeb6e042fbb5ad121e91de2860, d18126ddd7c840049afba377de0fa029; no voters:
I20260928 14:53:46.509088 28432 raft_consensus.cc:2781] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:46.509279 28432 raft_consensus.cc:686] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860 [term 1 LEADER]: Becoming Leader. State: Replica: c0148ddeb6e042fbb5ad121e91de2860, State: Running, Role: LEADER
I20260928 14:53:46.509645 28432 consensus_queue.cc:237] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18126ddd7c840049afba377de0fa029" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35679 } } peers { permanent_uuid: "c0148ddeb6e042fbb5ad121e91de2860" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 42409 } } peers { permanent_uuid: "dd1d5fa791f449278948c4abbb832e43" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35257 } }
I20260928 14:53:46.509719 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.509583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.131:55287)
0928 14:53:46.509663 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.509695 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 14:53:46.510390 27878 catalog_manager.cc:5495] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0148ddeb6e042fbb5ad121e91de2860 (127.20.116.131). New cstate: current_term: 1 leader_uuid: "c0148ddeb6e042fbb5ad121e91de2860" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18126ddd7c840049afba377de0fa029" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0148ddeb6e042fbb5ad121e91de2860" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 42409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd1d5fa791f449278948c4abbb832e43" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35257 } health_report { overall_health: UNKNOWN } } }
I20260928 14:53:46.517400 28426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.514286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50798 (local address 127.20.116.131:42409)
0928 14:53:46.514351 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:46.514353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:46.514361 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:53:46.514389 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:46.514390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:46.514399 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:46.514445 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:46.514645 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:46.515436 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:46.516631 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:46.516940 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:46.517010 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:46.517167 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:46.517195 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:46.517231 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:46.517286 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 14:53:46.517305 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:53:46.518271 28012 raft_consensus.cc:1261] T ac8f7073506248b8bb9501bc0cfcf918 P dd1d5fa791f449278948c4abbb832e43 [term 1 FOLLOWER]: Refusing update from remote peer c0148ddeb6e042fbb5ad121e91de2860: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:46.518271 28149 raft_consensus.cc:1261] T ac8f7073506248b8bb9501bc0cfcf918 P d18126ddd7c840049afba377de0fa029 [term 1 FOLLOWER]: Refusing update from remote peer c0148ddeb6e042fbb5ad121e91de2860: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:46.518517 28437 consensus_queue.cc:1035] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d18126ddd7c840049afba377de0fa029" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:46.518711 28432 consensus_queue.cc:1035] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd1d5fa791f449278948c4abbb832e43" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:46.519646 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.519526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.129:33095)
0928 14:53:46.519582 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.519613 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 14:53:46.519752 28408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.519537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.130:46477)
0928 14:53:46.519687 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.519720 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260928 14:53:46.520792 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.520642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.131:50535)
0928 14:53:46.520736 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.520764 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 14:53:46.520939 28442 mvcc.cc:205] Tried to move back new op lower bound from 7334327199816957952 to 7334327199784243200. Current Snapshot: MvccSnapshot[applied={T|T < 7334327199816957952}]
I20260928 14:53:46.520993 28408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.520912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.130:41045)
0928 14:53:46.520948 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.520972 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 14:53:46.521275 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.521135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.131:32787)
0928 14:53:46.521219 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.521245 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 14:53:46.521479 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.521383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35947 (local address 127.20.116.129:36905)
0928 14:53:46.521422 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.521446 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:35947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 14:53:46.521664 28441 mvcc.cc:205] Tried to move back new op lower bound from 7334327199816957952 to 7334327199784243200. Current Snapshot: MvccSnapshot[applied={T|T < 7334327199816957952}]
I20260928 14:53:46.522292 28440 mvcc.cc:205] Tried to move back new op lower bound from 7334327199816957952 to 7334327199784243200. Current Snapshot: MvccSnapshot[applied={T|T < 7334327199816957952 or (T in {7334327199816957952})}]
I20260928 14:53:46.522933 27877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:46.523031 27877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:46.524266 27877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62ebad5801fc4460bfc70a394730e0b1: Sending DeleteTablet for 3 replicas of tablet ac8f7073506248b8bb9501bc0cfcf918
I20260928 14:53:46.524559 28129 tablet_service.cc:1514] Processing DeleteTablet for tablet ac8f7073506248b8bb9501bc0cfcf918 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:46 UTC) from {username='oryx', principal='oryx/127.20.116.188@KRBTEST.COM'} at 127.0.0.1:53908
I20260928 14:53:46.524617 28265 tablet_service.cc:1514] Processing DeleteTablet for tablet ac8f7073506248b8bb9501bc0cfcf918 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:46 UTC) from {username='oryx', principal='oryx/127.20.116.188@KRBTEST.COM'} at 127.0.0.1:50782
I20260928 14:53:46.524652 27992 tablet_service.cc:1514] Processing DeleteTablet for tablet ac8f7073506248b8bb9501bc0cfcf918 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:46 UTC) from {username='oryx', principal='oryx/127.20.116.188@KRBTEST.COM'} at 127.0.0.1:50300
I20260928 14:53:46.524766 28448 tablet_replica.cc:330] T ac8f7073506248b8bb9501bc0cfcf918 P d18126ddd7c840049afba377de0fa029: stopping tablet replica
I20260928 14:53:46.524835 28448 raft_consensus.cc:2229] T ac8f7073506248b8bb9501bc0cfcf918 P d18126ddd7c840049afba377de0fa029 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:46.524871 28449 tablet_replica.cc:330] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860: stopping tablet replica
I20260928 14:53:46.524885 28448 raft_consensus.cc:2258] T ac8f7073506248b8bb9501bc0cfcf918 P d18126ddd7c840049afba377de0fa029 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:46.525007 28449 raft_consensus.cc:2229] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:53:46.525055 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 27836
I20260928 14:53:46.525091 28449 raft_consensus.cc:2258] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:46.525208 28448 ts_tablet_manager.cc:1905] T ac8f7073506248b8bb9501bc0cfcf918 P d18126ddd7c840049afba377de0fa029: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:46.525900 28449 ts_tablet_manager.cc:1905] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:46.527601 28448 ts_tablet_manager.cc:1918] T ac8f7073506248b8bb9501bc0cfcf918 P d18126ddd7c840049afba377de0fa029: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:53:46.527662 28448 log.cc:1196] T ac8f7073506248b8bb9501bc0cfcf918 P d18126ddd7c840049afba377de0fa029: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/ac8f7073506248b8bb9501bc0cfcf918
I20260928 14:53:46.527662 28449 ts_tablet_manager.cc:1918] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:53:46.527717 28449 log.cc:1196] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/ac8f7073506248b8bb9501bc0cfcf918
I20260928 14:53:46.527894 28448 ts_tablet_manager.cc:1939] T ac8f7073506248b8bb9501bc0cfcf918 P d18126ddd7c840049afba377de0fa029: Deleting consensus metadata
I20260928 14:53:46.527977 28449 ts_tablet_manager.cc:1939] T ac8f7073506248b8bb9501bc0cfcf918 P c0148ddeb6e042fbb5ad121e91de2860: Deleting consensus metadata
I20260928 14:53:46.528383 27865 catalog_manager.cc:4841] TS c0148ddeb6e042fbb5ad121e91de2860 (127.20.116.131:42409): tablet ac8f7073506248b8bb9501bc0cfcf918 (table test-table [id=52fa3b4ca61747f7a9604b0db5dfc3af]) successfully deleted
I20260928 14:53:46.528503 27865 catalog_manager.cc:4841] TS d18126ddd7c840049afba377de0fa029 (127.20.116.130:35679): tablet ac8f7073506248b8bb9501bc0cfcf918 (table test-table [id=52fa3b4ca61747f7a9604b0db5dfc3af]) successfully deleted
I20260928 14:53:46.532073 28425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:46.531930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:35257 (local address 127.0.0.1:50302)
0928 14:53:46.531993 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:46.532028 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.129:35257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 14:53:46.532054 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 28066
W20260928 14:53:46.532649 27866 catalog_manager.cc:4584] TS dd1d5fa791f449278948c4abbb832e43 (127.20.116.129:35257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ac8f7073506248b8bb9501bc0cfcf918: Network error: Client connection negotiation failed: client connection to 127.20.116.129:35257: connect: Connection refused (error 111)
I20260928 14:53:46.537681 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 28202
I20260928 14:53:46.543476 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 27553
I20260928 14:53:46.548599 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 27740
2026-09-28T14:53:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6417 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:53:46 dist-test-slave-pxl6 krb5kdc[28453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:53:46 dist-test-slave-pxl6 krb5kdc[28453](info): set up 2 sockets
Sep 28 14:53:46 dist-test-slave-pxl6 krb5kdc[28453](info): commencing operation
krb5kdc: starting...
W20260928 14:53:47.588181 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:53:47 dist-test-slave-pxl6 krb5kdc[28453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:53:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:53:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.190.keytab.
Entry for principal oryx/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.190.keytab.
I20260928 14:53:47.713410 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:37713
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.190.keytab
--principal=oryx/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:37713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:47.799324 28466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:47.799539 28466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:47.799562 28466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:47.800767 28466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:53:47.800817 28466 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 14:53:47.800832 28466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:47.800844 28466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:47.800856 28466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:53:47.800868 28466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:53:47.802431 28466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:37713
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.190.keytab
--principal=oryx/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:37713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:47.802775 28466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:47.803056 28466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:47.808502 28472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:47.808517 28471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:47.808521 28474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:47.808763 28466 server_base.cc:1034] running on GCE node
I20260928 14:53:47.808993 28466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:47.809245 28466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:47.810416 28466 hybrid_clock.cc:648] HybridClock initialized: now 1790607227810407 us; error 42 us; skew 500 ppm
Sep 28 14:53:47 dist-test-slave-pxl6 krb5kdc[28453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607227, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:47.812943 28466 init.cc:377] Logged in from keytab as oryx/127.20.116.190@KRBTEST.COM (short username oryx)
I20260928 14:53:47.813412 28466 webserver.cc:460] Webserver started at http://127.20.116.190:33243/ using document root <none> and password file <none>
I20260928 14:53:47.813645 28466 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:47.813719 28466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:47.813867 28466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:47.815002 28466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "5ac485d6034446998f660928ada1907b"
format_stamp: "Formatted at 2026-09-28 14:53:47 on dist-test-slave-pxl6"
server_key: "becda680605056628fbd05082523ecb7"
server_key_iv: "be2f3b920e8be5dc762f914c89c4b68e"
server_key_version: "encryptionkey@0"
I20260928 14:53:47.815382 28466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "5ac485d6034446998f660928ada1907b"
format_stamp: "Formatted at 2026-09-28 14:53:47 on dist-test-slave-pxl6"
server_key: "becda680605056628fbd05082523ecb7"
server_key_iv: "be2f3b920e8be5dc762f914c89c4b68e"
server_key_version: "encryptionkey@0"
I20260928 14:53:47.816816 28466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:53:47.817615 28481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:47.817823 28466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:47.817932 28466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "5ac485d6034446998f660928ada1907b"
format_stamp: "Formatted at 2026-09-28 14:53:47 on dist-test-slave-pxl6"
server_key: "becda680605056628fbd05082523ecb7"
server_key_iv: "be2f3b920e8be5dc762f914c89c4b68e"
server_key_version: "encryptionkey@0"
I20260928 14:53:47.818038 28466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:47.840263 28466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:47.842351 28466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:47.842549 28466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:47.847422 28466 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:37713
I20260928 14:53:47.847468 28543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:37713 every 8 connection(s)
I20260928 14:53:47.847829 28466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:53:47.848510 28544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:47.848903 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 28466
I20260928 14:53:47.849018 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:53:47.849238 20946 external_mini_cluster.cc:1428] Setting key 94e78caa4a7a7c48a5972f220f09c69d
I20260928 14:53:47.851436 28544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b: Bootstrap starting.
I20260928 14:53:47.852106 28544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:47.852404 28544 log.cc:824] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:47.853170 28544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b: No bootstrap required, opened a new log
I20260928 14:53:47.855890 28544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac485d6034446998f660928ada1907b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 37713 } }
I20260928 14:53:47.856030 28544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:47.856052 28544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ac485d6034446998f660928ada1907b, State: Initialized, Role: FOLLOWER
I20260928 14:53:47.856196 28544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac485d6034446998f660928ada1907b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 37713 } }
I20260928 14:53:47.856273 28544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:53:47.856323 28544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:53:47.856372 28544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b [term 0 FOLLOWER]: Advancing to term 1
Sep 28 14:53:47 dist-test-slave-pxl6 krb5kdc[28453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM
I20260928 14:53:47.857111 28544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac485d6034446998f660928ada1907b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 37713 } }
I20260928 14:53:47.857234 28544 leader_election.cc:302] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ac485d6034446998f660928ada1907b; no voters:
I20260928 14:53:47.857411 28544 leader_election.cc:288] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:53:47.857673 28544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:47.857846 28549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:47.858022 28549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b [term 1 LEADER]: Becoming Leader. State: Replica: 5ac485d6034446998f660928ada1907b, State: Running, Role: LEADER
I20260928 14:53:47.858657 28549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac485d6034446998f660928ada1907b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 37713 } }
I20260928 14:53:47.859191 28551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ac485d6034446998f660928ada1907b. Latest consensus state: current_term: 1 leader_uuid: "5ac485d6034446998f660928ada1907b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac485d6034446998f660928ada1907b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 37713 } } }
I20260928 14:53:47.859611 28550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ac485d6034446998f660928ada1907b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac485d6034446998f660928ada1907b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 37713 } } }
I20260928 14:53:47.859699 28550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:47.859287 28551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b [sys.catalog]: This master's current role is: LEADER
W20260928 14:53:47.860116 28563 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:53:47.860195 28563 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:53:47.860288 28564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:53:47.860519 28564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:53:47.861248 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:47.849802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55784 (local address 127.20.116.190:37713)
0928 14:53:47.850007 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:47.850024 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:47.850050 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:53:47.850473 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:47.850479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:47.850525 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:47.850734 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:47.851382 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:47.852421 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:47.853488 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:47.853910 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:47.853993 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:47.856966 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:47.856970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:47.856970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:47.856982 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:47.859153 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:47.859619 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:47.859622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:47.859622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:47.859734 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:47.859970 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:47.859971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:47.859971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:47.860299 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:47.860443 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:47.860891 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 14:53:47.861034 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":81,"threads_started":1}
I20260928 14:53:47.863020 28564 catalog_manager.cc:1347] Generated new cluster ID: 7157e901377546ff8c1d4eccb9137e43
I20260928 14:53:47.863073 28564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:53:47.874840 28564 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:53:47.875619 28564 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:53:47.883558 28564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b: Generated new TSK 0
I20260928 14:53:47.883790 28564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.129.keytab.
Entry for principal oryx/127.20.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.129.keytab.
I20260928 14:53:47.914310 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:37713
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.129.keytab
--principal=oryx/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:47.999993 28570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:48.000254 28570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:48.000299 28570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:48.001560 28570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:48.001644 28570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:48.001679 28570 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:53:48.001724 28570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:53:48.003419 28570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.129.keytab
--principal=oryx/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.116.190:37713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:48.003732 28570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:48.003993 28570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:48.009328 28578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:48.009334 28575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:48.009354 28576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:48.009639 28570 server_base.cc:1034] running on GCE node
I20260928 14:53:48.009943 28570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:48.010213 28570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:48.011363 28570 hybrid_clock.cc:648] HybridClock initialized: now 1790607228011351 us; error 35 us; skew 500 ppm
Sep 28 14:53:48 dist-test-slave-pxl6 krb5kdc[28453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607228, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:48.014117 28570 init.cc:377] Logged in from keytab as oryx/127.20.116.129@KRBTEST.COM (short username oryx)
I20260928 14:53:48.014647 28570 webserver.cc:460] Webserver started at http://127.20.116.129:35621/ using document root <none> and password file <none>
I20260928 14:53:48.014899 28570 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:48.014968 28570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:48.015110 28570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:48.016220 28570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "bf2ace9ae64b49d19d11dd8b759ded4e"
format_stamp: "Formatted at 2026-09-28 14:53:48 on dist-test-slave-pxl6"
server_key: "de7e402021e8ed056fabfadf30591a60"
server_key_iv: "7061e3ba7d878bd4fb2cc4bc2af609fd"
server_key_version: "encryptionkey@0"
I20260928 14:53:48.016614 28570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "bf2ace9ae64b49d19d11dd8b759ded4e"
format_stamp: "Formatted at 2026-09-28 14:53:48 on dist-test-slave-pxl6"
server_key: "de7e402021e8ed056fabfadf30591a60"
server_key_iv: "7061e3ba7d878bd4fb2cc4bc2af609fd"
server_key_version: "encryptionkey@0"
I20260928 14:53:48.018123 28570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:53:48.019042 28585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:48.019320 28570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:53:48.019440 28570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "bf2ace9ae64b49d19d11dd8b759ded4e"
format_stamp: "Formatted at 2026-09-28 14:53:48 on dist-test-slave-pxl6"
server_key: "de7e402021e8ed056fabfadf30591a60"
server_key_iv: "7061e3ba7d878bd4fb2cc4bc2af609fd"
server_key_version: "encryptionkey@0"
I20260928 14:53:48.019543 28570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:48.035342 28570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:48.037793 28570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:48.037971 28570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:48.038596 28570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:48.038655 28570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:48.038693 28570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:48.038738 28570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:53:48 dist-test-slave-pxl6 krb5kdc[28453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607228, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM
I20260928 14:53:48.046108 28570 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:34741
I20260928 14:53:48.046196 28700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:34741 every 8 connection(s)
I20260928 14:53:48.046589 28570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:53:48.049431 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.039244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:48131 (local address 127.20.116.190:37713)
0928 14:53:48.039317 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.039320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.039532 (+ 212us) server_negotiation.cc:407] Connection header received
0928 14:53:48.040115 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.040116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.040131 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.040215 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:48.040893 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.041773 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.043126 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.043507 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.043585 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.045925 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:48.045926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:48.045927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:48.045937 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:48.047877 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:48.048363 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:48.048365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:48.048365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:48.048436 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:48.048710 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:48.048711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:48.048712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:48.048852 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:48.048929 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:48.049230 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 14:53:48.049334 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:53:48.049659 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.039223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:37713 (local address 127.20.116.129:48131)
0928 14:53:48.039397 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:48.039438 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:48.039994 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:48.040322 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:48.040325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:48.040577 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:48.040787 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.040792 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.041881 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.041883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.043014 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.043017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.043631 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.043632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.043761 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.044298 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:48.044308 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:48.045821 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:48.047993 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:48.048002 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:48.048004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:48.048274 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:48.048510 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:48.048511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:48.048512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:48.048640 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:48.048930 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:48.048939 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:48.049089 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 14:53:48.049207 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 14:53:48.049385 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
I20260928 14:53:48.050268 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 28570
I20260928 14:53:48.050283 28701 heartbeater.cc:347] Connected to a master server at 127.20.116.190:37713
I20260928 14:53:48.050374 28701 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:48.050391 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:53:48.050629 28701 heartbeater.cc:511] Master 127.20.116.190:37713 requested a full tablet report, sending...
I20260928 14:53:48.050673 20946 external_mini_cluster.cc:1428] Setting key f4546a0a0bc2c72f4581d0f51a73304a
I20260928 14:53:48.051542 28498 ts_manager.cc:194] Registered new tserver with Master: bf2ace9ae64b49d19d11dd8b759ded4e (127.20.116.129:34741)
I20260928 14:53:48.052210 28498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:48131
I20260928 14:53:48.058996 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.051431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:43205 (local address 127.20.116.190:37713)
0928 14:53:48.051473 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.051475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.052572 (+ 1097us) server_negotiation.cc:407] Connection header received
0928 14:53:48.052674 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.052675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.052687 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.052731 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:48.053124 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.053596 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.054697 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.054903 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.054959 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.056091 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:48.056092 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:48.056092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:48.056098 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:48.057650 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:48.058082 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:48.058083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:48.058084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:48.058147 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:48.058389 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:48.058390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:48.058391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:48.058510 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:48.058636 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:48.058811 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 14:53:48.058900 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:53:48.059113 28707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.051421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:37713 (local address 127.20.116.129:43205)
0928 14:53:48.052491 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:48.052504 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:48.052570 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:48.052744 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:48.052745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:48.052961 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:48.053045 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.053047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.053702 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.053703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.054598 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.054601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.054981 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.054982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.055059 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.055543 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:48.055548 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:48.055998 (+ 450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:48.057785 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:48.057786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:48.057787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:48.057981 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:48.058221 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:48.058222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:48.058222 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:48.058319 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:48.058628 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:48.058629 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:48.058683 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 14:53:48.058770 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:53:48.058952 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"thread_start_us":65,"threads_started":1}
WARNING: no policy specified for oryx/127.20.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.130.keytab.
Entry for principal oryx/127.20.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.130.keytab.
I20260928 14:53:48.081333 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:37713
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.130.keytab
--principal=oryx/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:48.166945 28710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:48.167166 28710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:48.167191 28710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:48.168334 28710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:48.168381 28710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:48.168402 28710 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:53:48.168421 28710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:53:48.169982 28710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.130.keytab
--principal=oryx/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.116.190:37713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:48.170198 28710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:48.170390 28710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:48.175719 28718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:48.175828 28716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:48.175833 28715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:48.176162 28710 server_base.cc:1034] running on GCE node
I20260928 14:53:48.176409 28710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:48.176709 28710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:48.177904 28710 hybrid_clock.cc:648] HybridClock initialized: now 1790607228177878 us; error 48 us; skew 500 ppm
Sep 28 14:53:48 dist-test-slave-pxl6 krb5kdc[28453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607228, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:48.180823 28710 init.cc:377] Logged in from keytab as oryx/127.20.116.130@KRBTEST.COM (short username oryx)
I20260928 14:53:48.181298 28710 webserver.cc:460] Webserver started at http://127.20.116.130:42775/ using document root <none> and password file <none>
I20260928 14:53:48.181555 28710 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:48.181633 28710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:48.181783 28710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:48.182955 28710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "bfdbd4a7b6004d4db726b89031018cb8"
format_stamp: "Formatted at 2026-09-28 14:53:48 on dist-test-slave-pxl6"
server_key: "94ea724f1703831ca25ee6cf506d31cb"
server_key_iv: "2176fe19172392a1128fc496dfe5828e"
server_key_version: "encryptionkey@0"
I20260928 14:53:48.183327 28710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "bfdbd4a7b6004d4db726b89031018cb8"
format_stamp: "Formatted at 2026-09-28 14:53:48 on dist-test-slave-pxl6"
server_key: "94ea724f1703831ca25ee6cf506d31cb"
server_key_iv: "2176fe19172392a1128fc496dfe5828e"
server_key_version: "encryptionkey@0"
I20260928 14:53:48.184832 28710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:53:48.185837 28725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:48.186066 28710 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:48.186182 28710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "bfdbd4a7b6004d4db726b89031018cb8"
format_stamp: "Formatted at 2026-09-28 14:53:48 on dist-test-slave-pxl6"
server_key: "94ea724f1703831ca25ee6cf506d31cb"
server_key_iv: "2176fe19172392a1128fc496dfe5828e"
server_key_version: "encryptionkey@0"
I20260928 14:53:48.186264 28710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:48.202096 28710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:48.204773 28710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:48.204948 28710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:48.205607 28710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:48.205683 28710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:48.205734 28710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:48.205756 28710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:53:48 dist-test-slave-pxl6 krb5kdc[28453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607228, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM
I20260928 14:53:48.213028 28710 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:45141
I20260928 14:53:48.213092 28840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:45141 every 8 connection(s)
I20260928 14:53:48.213414 28710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:53:48.216464 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.206069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:42323 (local address 127.20.116.190:37713)
0928 14:53:48.206141 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.206144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.206354 (+ 210us) server_negotiation.cc:407] Connection header received
0928 14:53:48.206955 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.206956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.206971 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.207056 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:48.207901 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.208736 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.210364 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.210757 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.210833 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.213398 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:48.213400 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:48.213400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:48.213409 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:48.214992 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:48.215440 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:48.215441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:48.215442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:48.215510 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:48.215774 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:48.215775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:48.215775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:48.215905 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:48.215985 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:48.216282 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 14:53:48.216368 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:53:48.216689 28734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.206057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:37713 (local address 127.20.116.130:42323)
0928 14:53:48.206203 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:48.206256 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:48.206848 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:48.207121 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:48.207125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:48.207430 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:48.207787 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.207795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.208831 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.208832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.210276 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.210279 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.210844 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.210845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.210994 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.211649 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:48.211661 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:48.213293 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:48.215098 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:48.215106 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:48.215108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:48.215348 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:48.215584 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:48.215585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:48.215585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:48.215702 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:48.215974 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:48.215981 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:48.216131 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 14:53:48.216258 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 14:53:48.216430 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260928 14:53:48.216890 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 28710
I20260928 14:53:48.216998 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:53:48.217141 28841 heartbeater.cc:347] Connected to a master server at 127.20.116.190:37713
I20260928 14:53:48.217231 28841 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:48.217238 20946 external_mini_cluster.cc:1428] Setting key bec058653d29a9368874cce57a471be1
I20260928 14:53:48.217389 28841 heartbeater.cc:511] Master 127.20.116.190:37713 requested a full tablet report, sending...
I20260928 14:53:48.218021 28497 ts_manager.cc:194] Registered new tserver with Master: bfdbd4a7b6004d4db726b89031018cb8 (127.20.116.130:45141)
I20260928 14:53:48.218580 28497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:42323
I20260928 14:53:48.226306 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.219608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:38361 (local address 127.20.116.190:37713)
0928 14:53:48.219677 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.219679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.219688 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:53:48.219717 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.219718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.219729 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.219792 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:48.220137 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.220835 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.221984 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.222229 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.222286 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.223616 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:48.223617 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:48.223617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:48.223624 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:48.225107 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:48.225519 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:48.225520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:48.225521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:48.225573 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:48.225782 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:48.225783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:48.225783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:48.225921 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:48.225975 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:48.226170 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 14:53:48.226240 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:53:48.226438 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.218182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:37713 (local address 127.20.116.130:38361)
0928 14:53:48.218316 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:48.218326 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:48.218366 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:48.219814 (+ 1448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:48.219815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:48.219967 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:48.220049 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.220051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.220909 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.220910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.221862 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.221865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.222315 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.222317 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.222402 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.223016 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:48.223022 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:48.223527 (+ 505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:48.225196 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:48.225198 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:48.225198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:48.225438 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:48.225640 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:48.225641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:48.225642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:48.225713 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:48.225991 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:48.225992 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:48.226064 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 14:53:48.226147 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 14:53:48.226274 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for oryx/127.20.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.131.keytab.
Entry for principal oryx/127.20.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.131.keytab.
I20260928 14:53:48.246073 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:37713
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.131.keytab
--principal=oryx/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:48.330327 28850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:48.330593 28850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:48.330636 28850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:48.331838 28850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:48.331908 28850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:48.331943 28850 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:53:48.331984 28850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:53:48.333513 28850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.131.keytab
--principal=oryx/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.116.190:37713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:48.333796 28850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:48.334043 28850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:48.339236 28856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:48.339257 28855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:48.339293 28858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:48.339706 28850 server_base.cc:1034] running on GCE node
I20260928 14:53:48.339989 28850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:48.340235 28850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:48.341406 28850 hybrid_clock.cc:648] HybridClock initialized: now 1790607228341392 us; error 26 us; skew 500 ppm
Sep 28 14:53:48 dist-test-slave-pxl6 krb5kdc[28453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607228, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:48.344101 28850 init.cc:377] Logged in from keytab as oryx/127.20.116.131@KRBTEST.COM (short username oryx)
I20260928 14:53:48.344553 28850 webserver.cc:460] Webserver started at http://127.20.116.131:37321/ using document root <none> and password file <none>
I20260928 14:53:48.344741 28850 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:48.344784 28850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:48.344871 28850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:48.345882 28850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "19a78e3744bb4e3cbb12de42595495ac"
format_stamp: "Formatted at 2026-09-28 14:53:48 on dist-test-slave-pxl6"
server_key: "833a77f298099e6ac3ca08989e6a55f4"
server_key_iv: "cd591b3144b1e2bd8e5f246904e05d46"
server_key_version: "encryptionkey@0"
I20260928 14:53:48.346195 28850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "19a78e3744bb4e3cbb12de42595495ac"
format_stamp: "Formatted at 2026-09-28 14:53:48 on dist-test-slave-pxl6"
server_key: "833a77f298099e6ac3ca08989e6a55f4"
server_key_iv: "cd591b3144b1e2bd8e5f246904e05d46"
server_key_version: "encryptionkey@0"
I20260928 14:53:48.347535 28850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:53:48.348260 28865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:48.348433 28850 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:48.348539 28850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "19a78e3744bb4e3cbb12de42595495ac"
format_stamp: "Formatted at 2026-09-28 14:53:48 on dist-test-slave-pxl6"
server_key: "833a77f298099e6ac3ca08989e6a55f4"
server_key_iv: "cd591b3144b1e2bd8e5f246904e05d46"
server_key_version: "encryptionkey@0"
I20260928 14:53:48.348615 28850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:48.369634 28850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:48.372323 28850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:48.372496 28850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:48.373078 28850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:48.373152 28850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:48.373198 28850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:48.373225 28850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:48.380226 28850 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:36181
Sep 28 14:53:48 dist-test-slave-pxl6 krb5kdc[28453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607228, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM
I20260928 14:53:48.380298 28980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:36181 every 8 connection(s)
I20260928 14:53:48.380664 28850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:53:48.381475 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 28850
I20260928 14:53:48.381608 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:53:48.381819 20946 external_mini_cluster.cc:1428] Setting key a9105dd8b223b440e9e022b2b4407fde
I20260928 14:53:48.383764 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.373598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:59231 (local address 127.20.116.190:37713)
0928 14:53:48.373675 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.373678 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.373846 (+ 168us) server_negotiation.cc:407] Connection header received
0928 14:53:48.374362 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.374363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.374378 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.374460 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:48.375270 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.376121 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.377813 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.378140 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.378214 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.380877 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:48.380879 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:48.380879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:48.380889 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:48.382396 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:48.382886 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:48.382887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:48.382887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:48.382941 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:48.383155 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:48.383156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:48.383156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:48.383273 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:48.383328 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:48.383625 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 14:53:48.383696 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:53:48.384079 28874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.373557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:37713 (local address 127.20.116.131:59231)
0928 14:53:48.373709 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:48.373755 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:48.374261 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:48.374592 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:48.374597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:48.374906 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:48.375121 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.375126 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.376230 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.376231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.377725 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.377728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.378230 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.378230 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.378374 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.379037 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:48.379048 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:48.380759 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:48.382458 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:48.382464 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:48.382467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:48.382818 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:48.383000 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:48.383001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:48.383001 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:48.383095 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:48.383342 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:48.383348 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:48.383488 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 14:53:48.383571 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 14:53:48.383838 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260928 14:53:48.384581 28981 heartbeater.cc:347] Connected to a master server at 127.20.116.190:37713
I20260928 14:53:48.384677 28981 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:48.384895 28981 heartbeater.cc:511] Master 127.20.116.190:37713 requested a full tablet report, sending...
I20260928 14:53:48.385254 28498 ts_manager.cc:194] Registered new tserver with Master: 19a78e3744bb4e3cbb12de42595495ac (127.20.116.131:36181)
I20260928 14:53:48.385772 28498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:59231
I20260928 14:53:48.390931 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.385732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:52211 (local address 127.20.116.190:37713)
0928 14:53:48.385773 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.385775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.385828 (+ 53us) server_negotiation.cc:407] Connection header received
0928 14:53:48.385924 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.385925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.385938 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.386008 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:48.386387 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.386931 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.387590 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.387765 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.387813 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.388608 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:48.388610 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:48.388610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:48.388619 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:48.389869 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:48.390237 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:48.390239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:48.390239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:48.390310 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:48.390491 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:48.390492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:48.390492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:48.390648 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:48.390715 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:48.390811 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 14:53:48.390871 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:53:48.390938 28987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.385629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:37713 (local address 127.20.116.131:52211)
0928 14:53:48.385778 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:48.385789 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:48.385843 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:48.386020 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:48.386020 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:48.386232 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:48.386312 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.386315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.387016 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.387017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.387521 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.387522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.387827 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.387828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.387875 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.388267 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:48.388271 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:48.388530 (+ 259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:48.389937 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:48.389937 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:48.389938 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:48.390093 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:48.390377 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:48.390377 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:48.390378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:48.390429 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:48.390705 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:48.390706 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:48.390743 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 14:53:48.390792 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 14:53:48.390867 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260928 14:53:48.393666 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:53:48 dist-test-slave-pxl6 krb5kdc[28453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607228, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:53:48 dist-test-slave-pxl6 krb5kdc[28453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:48 dist-test-slave-pxl6 krb5kdc[28453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:53:48.414657 28547 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:48.409085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.20.116.190:37713)
0928 14:53:48.409152 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.409154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.409164 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:53:48.409194 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.409195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.409213 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.409283 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:48.409848 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.410352 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.411201 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.411398 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.411477 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.414532 (+ 3055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60562: BlockingRecv error: recv got EOF from 127.0.0.1:60562 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 14:53:48 dist-test-slave-pxl6 krb5kdc[28453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607228, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM
I20260928 14:53:48.422315 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.415595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60576 (local address 127.20.116.190:37713)
0928 14:53:48.415673 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.415675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.415687 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:53:48.415768 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.415769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.415778 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.415828 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:48.416262 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.416739 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.417440 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.417622 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.417677 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.419937 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:48.419938 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:48.419938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:48.419945 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:48.421242 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:48.421543 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:48.421544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:48.421544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:48.421594 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:48.421822 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:48.421822 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:48.421823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:48.421932 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:48.422058 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:48.422139 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 14:53:48.422210 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:53:48.424156 28498 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:53:48.426642 28498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:53:48.433651 29005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.429231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57758 (local address 127.20.116.130:45141)
0928 14:53:48.429574 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.429579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.429592 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:53:48.429636 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.429638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.429655 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.429734 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:48.430003 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.431239 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.432358 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.433135 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.433247 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.433315 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:48.433499 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 14:53:48.433554 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":106,"threads_started":1}
I20260928 14:53:48.433627 29007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.429108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:45141 (local address 127.0.0.1:57758)
0928 14:53:48.429460 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:48.429471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:48.429512 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:48.429825 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:48.429828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:48.429831 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:48.429904 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.429906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.431324 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.431325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.432266 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.432268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.433258 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.433258 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.433372 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.433390 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 14:53:48.433474 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 14:53:48.433542 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":24,"thread_start_us":104,"threads_started":1}
I20260928 14:53:48.433733 29003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.429334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:36181 (local address 127.0.0.1:44284)
0928 14:53:48.429462 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:48.429473 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:48.429527 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:48.429968 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:48.429969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:48.429971 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:48.430025 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.430027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.431073 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.431074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.432394 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.432397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.433256 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.433257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.433372 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.433390 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 14:53:48.433474 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 14:53:48.433602 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":89,"thread_start_us":76,"threads_started":1}
I20260928 14:53:48.433971 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.429597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44284 (local address 127.20.116.131:36181)
0928 14:53:48.429768 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.429775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.429789 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:53:48.429844 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.429846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.429862 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.430122 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:48.430184 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.430967 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.432485 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.433167 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.433254 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.433315 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:48.433822 (+ 507us) server_negotiation.cc:299] Negotiation successful
0928 14:53:48.433881 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
I20260928 14:53:48.434494 28915 tablet_service.cc:1467] Processing CreateTablet for tablet 7817aaab3f35455aa9b3bc7c3a3c185f (DEFAULT_TABLE table=test-table [id=281d4c9ab68e4dcdb3b3ae6ff2667d77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:48.434473 28775 tablet_service.cc:1467] Processing CreateTablet for tablet 7817aaab3f35455aa9b3bc7c3a3c185f (DEFAULT_TABLE table=test-table [id=281d4c9ab68e4dcdb3b3ae6ff2667d77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:48.434907 28775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7817aaab3f35455aa9b3bc7c3a3c185f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:48.434907 28915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7817aaab3f35455aa9b3bc7c3a3c185f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:48.435057 29004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.429105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:34741 (local address 127.0.0.1:43376)
0928 14:53:48.429308 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:48.429354 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:48.429422 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:48.429856 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:48.429857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:48.429859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:48.429926 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.429929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.431293 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.431294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.432398 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.432399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.434815 (+ 2416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.434816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.434890 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.434894 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:53:48.434919 (+ 25us) client_negotiation.cc:240] Negotiation successful
0928 14:53:48.434949 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":7,"thread_start_us":81,"threads_started":1}
I20260928 14:53:48.435461 29006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.429296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43376 (local address 127.20.116.129:34741)
0928 14:53:48.429692 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.429696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.429705 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:53:48.429740 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.429742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.429754 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.429844 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:48.430095 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.431202 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.433353 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.434696 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.435266 (+ 570us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.435338 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:48.435361 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 14:53:48.435391 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":113,"threads_started":1}
I20260928 14:53:48.438946 29009 tablet_bootstrap.cc:492] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac: Bootstrap starting.
I20260928 14:53:48.439293 28635 tablet_service.cc:1467] Processing CreateTablet for tablet 7817aaab3f35455aa9b3bc7c3a3c185f (DEFAULT_TABLE table=test-table [id=281d4c9ab68e4dcdb3b3ae6ff2667d77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:48.439677 28635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7817aaab3f35455aa9b3bc7c3a3c185f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:48.439733 29009 tablet_bootstrap.cc:654] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:48.440047 29009 log.cc:824] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:48.440423 29010 tablet_bootstrap.cc:492] T 7817aaab3f35455aa9b3bc7c3a3c185f P bfdbd4a7b6004d4db726b89031018cb8: Bootstrap starting.
I20260928 14:53:48.440825 29009 tablet_bootstrap.cc:492] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac: No bootstrap required, opened a new log
I20260928 14:53:48.440915 29009 ts_tablet_manager.cc:1397] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:53:48.441460 29010 tablet_bootstrap.cc:654] T 7817aaab3f35455aa9b3bc7c3a3c185f P bfdbd4a7b6004d4db726b89031018cb8: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:48.441790 29010 log.cc:824] T 7817aaab3f35455aa9b3bc7c3a3c185f P bfdbd4a7b6004d4db726b89031018cb8: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:48.442719 29010 tablet_bootstrap.cc:492] T 7817aaab3f35455aa9b3bc7c3a3c185f P bfdbd4a7b6004d4db726b89031018cb8: No bootstrap required, opened a new log
I20260928 14:53:48.442801 29010 ts_tablet_manager.cc:1397] T 7817aaab3f35455aa9b3bc7c3a3c185f P bfdbd4a7b6004d4db726b89031018cb8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:53:48.442684 29009 raft_consensus.cc:348] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.442852 29009 raft_consensus.cc:374] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:48.442893 29009 raft_consensus.cc:729] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a78e3744bb4e3cbb12de42595495ac, State: Initialized, Role: FOLLOWER
I20260928 14:53:48.443008 29009 consensus_queue.cc:260] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.443277 29009 ts_tablet_manager.cc:1428] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 14:53:48.443409 28981 heartbeater.cc:503] Master 127.20.116.190:37713 was elected leader, sending a full tablet report...
I20260928 14:53:48.444095 29014 tablet_bootstrap.cc:492] T 7817aaab3f35455aa9b3bc7c3a3c185f P bf2ace9ae64b49d19d11dd8b759ded4e: Bootstrap starting.
I20260928 14:53:48.444728 29014 tablet_bootstrap.cc:654] T 7817aaab3f35455aa9b3bc7c3a3c185f P bf2ace9ae64b49d19d11dd8b759ded4e: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:48.445056 29014 log.cc:824] T 7817aaab3f35455aa9b3bc7c3a3c185f P bf2ace9ae64b49d19d11dd8b759ded4e: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:48.444983 29010 raft_consensus.cc:348] T 7817aaab3f35455aa9b3bc7c3a3c185f P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.445155 29010 raft_consensus.cc:374] T 7817aaab3f35455aa9b3bc7c3a3c185f P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:48.445202 29010 raft_consensus.cc:729] T 7817aaab3f35455aa9b3bc7c3a3c185f P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfdbd4a7b6004d4db726b89031018cb8, State: Initialized, Role: FOLLOWER
I20260928 14:53:48.445333 29010 consensus_queue.cc:260] T 7817aaab3f35455aa9b3bc7c3a3c185f P bfdbd4a7b6004d4db726b89031018cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.445616 29010 ts_tablet_manager.cc:1428] T 7817aaab3f35455aa9b3bc7c3a3c185f P bfdbd4a7b6004d4db726b89031018cb8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:53:48.445652 28841 heartbeater.cc:503] Master 127.20.116.190:37713 was elected leader, sending a full tablet report...
I20260928 14:53:48.445880 29014 tablet_bootstrap.cc:492] T 7817aaab3f35455aa9b3bc7c3a3c185f P bf2ace9ae64b49d19d11dd8b759ded4e: No bootstrap required, opened a new log
I20260928 14:53:48.445948 29014 ts_tablet_manager.cc:1397] T 7817aaab3f35455aa9b3bc7c3a3c185f P bf2ace9ae64b49d19d11dd8b759ded4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:53:48.447619 29014 raft_consensus.cc:348] T 7817aaab3f35455aa9b3bc7c3a3c185f P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.447750 29014 raft_consensus.cc:374] T 7817aaab3f35455aa9b3bc7c3a3c185f P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:48.447772 29014 raft_consensus.cc:729] T 7817aaab3f35455aa9b3bc7c3a3c185f P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2ace9ae64b49d19d11dd8b759ded4e, State: Initialized, Role: FOLLOWER
I20260928 14:53:48.447853 29014 consensus_queue.cc:260] T 7817aaab3f35455aa9b3bc7c3a3c185f P bf2ace9ae64b49d19d11dd8b759ded4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.448037 29014 ts_tablet_manager.cc:1428] T 7817aaab3f35455aa9b3bc7c3a3c185f P bf2ace9ae64b49d19d11dd8b759ded4e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:53:48.448115 28701 heartbeater.cc:503] Master 127.20.116.190:37713 was elected leader, sending a full tablet report...
W20260928 14:53:48.464084 28842 tablet.cc:2369] T 7817aaab3f35455aa9b3bc7c3a3c185f P bfdbd4a7b6004d4db726b89031018cb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:53:48.529605 29013 raft_consensus.cc:482] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:53:48.529749 29013 raft_consensus.cc:504] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.530158 29013 leader_election.cc:288] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf2ace9ae64b49d19d11dd8b759ded4e (127.20.116.129:34741), bfdbd4a7b6004d4db726b89031018cb8 (127.20.116.130:45141)
I20260928 14:53:48.533253 29005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.530513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:35909 (local address 127.20.116.130:45141)
0928 14:53:48.530581 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.530583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.530803 (+ 220us) server_negotiation.cc:407] Connection header received
0928 14:53:48.530858 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.530859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.530871 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.530926 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:48.531115 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.531594 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.532516 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.532915 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.532992 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.533000 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:48.533137 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 14:53:48.533176 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:53:48.533262 29006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.530461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:33123 (local address 127.20.116.129:34741)
0928 14:53:48.530511 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.530513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.530547 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:53:48.530645 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.530646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.530655 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.530732 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:48.530932 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.531488 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.532473 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.532852 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.532907 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.532914 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:48.533130 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 14:53:48.533168 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:53:48.533310 28874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.530403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:34741 (local address 127.20.116.131:33123)
0928 14:53:48.530460 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:48.530473 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:48.530521 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:48.530742 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:48.530743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:48.530746 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:48.530822 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.530825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.531577 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.531577 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.532375 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.532378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.532938 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.532939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.533051 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.533055 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:53:48.533123 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 14:53:48.533164 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 14:53:48.533322 29018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.530405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:45141 (local address 127.20.116.131:35909)
0928 14:53:48.530738 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:48.530749 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:48.530802 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:48.530949 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:48.530950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:48.530964 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:48.531025 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.531027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.531697 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.531698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.532400 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.532402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.533000 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.533001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.533053 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.533056 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:53:48.533122 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 14:53:48.533162 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":59,"thread_start_us":201,"threads_started":1}
I20260928 14:53:48.533660 28655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7817aaab3f35455aa9b3bc7c3a3c185f" candidate_uuid: "19a78e3744bb4e3cbb12de42595495ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" is_pre_election: true
I20260928 14:53:48.533838 28655 raft_consensus.cc:2454] T 7817aaab3f35455aa9b3bc7c3a3c185f P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19a78e3744bb4e3cbb12de42595495ac in term 0.
I20260928 14:53:48.533936 28795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7817aaab3f35455aa9b3bc7c3a3c185f" candidate_uuid: "19a78e3744bb4e3cbb12de42595495ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfdbd4a7b6004d4db726b89031018cb8" is_pre_election: true
I20260928 14:53:48.534108 28866 leader_election.cc:302] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19a78e3744bb4e3cbb12de42595495ac, bf2ace9ae64b49d19d11dd8b759ded4e; no voters:
I20260928 14:53:48.534149 28795 raft_consensus.cc:2454] T 7817aaab3f35455aa9b3bc7c3a3c185f P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19a78e3744bb4e3cbb12de42595495ac in term 0.
I20260928 14:53:48.534260 29013 raft_consensus.cc:2781] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:53:48.534399 29013 raft_consensus.cc:482] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:53:48.534416 29013 raft_consensus.cc:3037] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:48.535378 29013 raft_consensus.cc:504] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.535595 29013 leader_election.cc:288] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac [CANDIDATE]: Term 1 election: Requested vote from peers bf2ace9ae64b49d19d11dd8b759ded4e (127.20.116.129:34741), bfdbd4a7b6004d4db726b89031018cb8 (127.20.116.130:45141)
I20260928 14:53:48.535683 28795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7817aaab3f35455aa9b3bc7c3a3c185f" candidate_uuid: "19a78e3744bb4e3cbb12de42595495ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfdbd4a7b6004d4db726b89031018cb8"
I20260928 14:53:48.535720 28655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7817aaab3f35455aa9b3bc7c3a3c185f" candidate_uuid: "19a78e3744bb4e3cbb12de42595495ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e"
I20260928 14:53:48.535758 28795 raft_consensus.cc:3037] T 7817aaab3f35455aa9b3bc7c3a3c185f P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:48.535800 28655 raft_consensus.cc:3037] T 7817aaab3f35455aa9b3bc7c3a3c185f P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:48.536423 28795 raft_consensus.cc:2454] T 7817aaab3f35455aa9b3bc7c3a3c185f P bfdbd4a7b6004d4db726b89031018cb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19a78e3744bb4e3cbb12de42595495ac in term 1.
I20260928 14:53:48.536479 28655 raft_consensus.cc:2454] T 7817aaab3f35455aa9b3bc7c3a3c185f P bf2ace9ae64b49d19d11dd8b759ded4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19a78e3744bb4e3cbb12de42595495ac in term 1.
I20260928 14:53:48.536646 28869 leader_election.cc:302] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19a78e3744bb4e3cbb12de42595495ac, bfdbd4a7b6004d4db726b89031018cb8; no voters:
I20260928 14:53:48.536756 29013 raft_consensus.cc:2781] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:48.536936 29013 raft_consensus.cc:686] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac [term 1 LEADER]: Becoming Leader. State: Replica: 19a78e3744bb4e3cbb12de42595495ac, State: Running, Role: LEADER
I20260928 14:53:48.537053 29013 consensus_queue.cc:237] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.537993 28497 catalog_manager.cc:5495] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 19a78e3744bb4e3cbb12de42595495ac (127.20.116.131). New cstate: current_term: 1 leader_uuid: "19a78e3744bb4e3cbb12de42595495ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } health_report { overall_health: HEALTHY } } }
W20260928 14:53:48.547473 28702 tablet.cc:2369] T 7817aaab3f35455aa9b3bc7c3a3c185f P bf2ace9ae64b49d19d11dd8b759ded4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:53:48.550158 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.547819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44296 (local address 127.20.116.131:36181)
0928 14:53:48.547880 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.547883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.547895 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:53:48.547935 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.547935 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.547952 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.548029 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:48.548152 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.548623 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.549354 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.549556 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.549615 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.549750 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:48.549849 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:48.549910 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:48.550075 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 14:53:48.550096 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:53:48.552119 28655 raft_consensus.cc:1261] T 7817aaab3f35455aa9b3bc7c3a3c185f P bf2ace9ae64b49d19d11dd8b759ded4e [term 1 FOLLOWER]: Refusing update from remote peer 19a78e3744bb4e3cbb12de42595495ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:48.552462 28795 raft_consensus.cc:1261] T 7817aaab3f35455aa9b3bc7c3a3c185f P bfdbd4a7b6004d4db726b89031018cb8 [term 1 FOLLOWER]: Refusing update from remote peer 19a78e3744bb4e3cbb12de42595495ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:48.552522 29013 consensus_queue.cc:1035] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:48.552747 29013 consensus_queue.cc:1035] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:48.555990 29024 mvcc.cc:205] Tried to move back new op lower bound from 7334327208146821120 to 7334327208088657920. Current Snapshot: MvccSnapshot[applied={T|T < 7334327208146821120}]
I20260928 14:53:48.557793 28498 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:48.557895 28498 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:48.558732 28498 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b: Sending DeleteTablet for 3 replicas of tablet 7817aaab3f35455aa9b3bc7c3a3c185f
I20260928 14:53:48.559103 28635 tablet_service.cc:1514] Processing DeleteTablet for tablet 7817aaab3f35455aa9b3bc7c3a3c185f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:48 UTC) from {username='oryx', principal='oryx/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:43376
I20260928 14:53:48.559141 28915 tablet_service.cc:1514] Processing DeleteTablet for tablet 7817aaab3f35455aa9b3bc7c3a3c185f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:48 UTC) from {username='oryx', principal='oryx/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:44284
I20260928 14:53:48.559341 28775 tablet_service.cc:1514] Processing DeleteTablet for tablet 7817aaab3f35455aa9b3bc7c3a3c185f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:48 UTC) from {username='oryx', principal='oryx/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:57758
I20260928 14:53:48.559474 28498 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 14:53:48.559641 29033 tablet_replica.cc:330] T 7817aaab3f35455aa9b3bc7c3a3c185f P bfdbd4a7b6004d4db726b89031018cb8: stopping tablet replica
W20260928 14:53:48.559667 28498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:53:48.559743 29033 raft_consensus.cc:2229] T 7817aaab3f35455aa9b3bc7c3a3c185f P bfdbd4a7b6004d4db726b89031018cb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:48.560138 29032 tablet_replica.cc:330] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac: stopping tablet replica
I20260928 14:53:48.560243 29032 raft_consensus.cc:2229] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:53:48.560372 29032 raft_consensus.cc:2258] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:48.560632 29033 raft_consensus.cc:2258] T 7817aaab3f35455aa9b3bc7c3a3c185f P bfdbd4a7b6004d4db726b89031018cb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:48.561151 28635 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2db9382a894766af65c470df334a82 (DEFAULT_TABLE table=test-table [id=a91b0c8245a14495bbc8d91c52a92f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:48.561223 28915 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2db9382a894766af65c470df334a82 (DEFAULT_TABLE table=test-table [id=a91b0c8245a14495bbc8d91c52a92f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:48.561261 28775 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2db9382a894766af65c470df334a82 (DEFAULT_TABLE table=test-table [id=a91b0c8245a14495bbc8d91c52a92f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:48.561296 28635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b2db9382a894766af65c470df334a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:48.561343 28915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b2db9382a894766af65c470df334a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:48.561357 28775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b2db9382a894766af65c470df334a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:48.562727 29014 tablet_bootstrap.cc:492] T 2b2db9382a894766af65c470df334a82 P bf2ace9ae64b49d19d11dd8b759ded4e: Bootstrap starting.
I20260928 14:53:48.562731 29010 tablet_bootstrap.cc:492] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8: Bootstrap starting.
I20260928 14:53:48.562745 29009 tablet_bootstrap.cc:492] T 2b2db9382a894766af65c470df334a82 P 19a78e3744bb4e3cbb12de42595495ac: Bootstrap starting.
I20260928 14:53:48.563083 29014 tablet_bootstrap.cc:654] T 2b2db9382a894766af65c470df334a82 P bf2ace9ae64b49d19d11dd8b759ded4e: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:48.563195 29010 tablet_bootstrap.cc:654] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:48.563195 29009 tablet_bootstrap.cc:654] T 2b2db9382a894766af65c470df334a82 P 19a78e3744bb4e3cbb12de42595495ac: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:48.563648 29032 ts_tablet_manager.cc:1905] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:48.563730 29014 tablet_bootstrap.cc:492] T 2b2db9382a894766af65c470df334a82 P bf2ace9ae64b49d19d11dd8b759ded4e: No bootstrap required, opened a new log
I20260928 14:53:48.563778 29014 ts_tablet_manager.cc:1397] T 2b2db9382a894766af65c470df334a82 P bf2ace9ae64b49d19d11dd8b759ded4e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:48.563934 29014 raft_consensus.cc:348] T 2b2db9382a894766af65c470df334a82 P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.564028 29014 raft_consensus.cc:374] T 2b2db9382a894766af65c470df334a82 P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:48.564055 29014 raft_consensus.cc:729] T 2b2db9382a894766af65c470df334a82 P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2ace9ae64b49d19d11dd8b759ded4e, State: Initialized, Role: FOLLOWER
I20260928 14:53:48.564064 29031 tablet_replica.cc:330] T 7817aaab3f35455aa9b3bc7c3a3c185f P bf2ace9ae64b49d19d11dd8b759ded4e: stopping tablet replica
I20260928 14:53:48.564098 29014 consensus_queue.cc:260] T 2b2db9382a894766af65c470df334a82 P bf2ace9ae64b49d19d11dd8b759ded4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.564193 29031 raft_consensus.cc:2229] T 7817aaab3f35455aa9b3bc7c3a3c185f P bf2ace9ae64b49d19d11dd8b759ded4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:48.564211 29010 tablet_bootstrap.cc:492] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8: No bootstrap required, opened a new log
I20260928 14:53:48.564219 29014 ts_tablet_manager.cc:1428] T 2b2db9382a894766af65c470df334a82 P bf2ace9ae64b49d19d11dd8b759ded4e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:53:48.564248 29010 ts_tablet_manager.cc:1397] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:53:48.564309 29031 raft_consensus.cc:2258] T 7817aaab3f35455aa9b3bc7c3a3c185f P bf2ace9ae64b49d19d11dd8b759ded4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:48.564409 29010 raft_consensus.cc:348] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.564476 29010 raft_consensus.cc:374] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:48.564500 29010 raft_consensus.cc:729] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfdbd4a7b6004d4db726b89031018cb8, State: Initialized, Role: FOLLOWER
I20260928 14:53:48.564554 29010 consensus_queue.cc:260] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.564638 29010 ts_tablet_manager.cc:1428] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:48.564775 29033 ts_tablet_manager.cc:1905] T 7817aaab3f35455aa9b3bc7c3a3c185f P bfdbd4a7b6004d4db726b89031018cb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:48.566134 29032 ts_tablet_manager.cc:1918] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:53:48.566217 29032 log.cc:1196] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/7817aaab3f35455aa9b3bc7c3a3c185f
I20260928 14:53:48.566406 29032 ts_tablet_manager.cc:1939] T 7817aaab3f35455aa9b3bc7c3a3c185f P 19a78e3744bb4e3cbb12de42595495ac: Deleting consensus metadata
I20260928 14:53:48.566695 29033 ts_tablet_manager.cc:1918] T 7817aaab3f35455aa9b3bc7c3a3c185f P bfdbd4a7b6004d4db726b89031018cb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:53:48.566766 29031 ts_tablet_manager.cc:1905] T 7817aaab3f35455aa9b3bc7c3a3c185f P bf2ace9ae64b49d19d11dd8b759ded4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:48.566874 29033 log.cc:1196] T 7817aaab3f35455aa9b3bc7c3a3c185f P bfdbd4a7b6004d4db726b89031018cb8: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/7817aaab3f35455aa9b3bc7c3a3c185f
I20260928 14:53:48.566881 28485 catalog_manager.cc:4841] TS 19a78e3744bb4e3cbb12de42595495ac (127.20.116.131:36181): tablet 7817aaab3f35455aa9b3bc7c3a3c185f (table test-table [id=281d4c9ab68e4dcdb3b3ae6ff2667d77]) successfully deleted
I20260928 14:53:48.567219 29033 ts_tablet_manager.cc:1939] T 7817aaab3f35455aa9b3bc7c3a3c185f P bfdbd4a7b6004d4db726b89031018cb8: Deleting consensus metadata
I20260928 14:53:48.567756 28485 catalog_manager.cc:4841] TS bfdbd4a7b6004d4db726b89031018cb8 (127.20.116.130:45141): tablet 7817aaab3f35455aa9b3bc7c3a3c185f (table test-table [id=281d4c9ab68e4dcdb3b3ae6ff2667d77]) successfully deleted
I20260928 14:53:48.568506 29031 ts_tablet_manager.cc:1918] T 7817aaab3f35455aa9b3bc7c3a3c185f P bf2ace9ae64b49d19d11dd8b759ded4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:53:48.568591 29031 log.cc:1196] T 7817aaab3f35455aa9b3bc7c3a3c185f P bf2ace9ae64b49d19d11dd8b759ded4e: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-0/wal/wals/7817aaab3f35455aa9b3bc7c3a3c185f
I20260928 14:53:48.568667 29009 tablet_bootstrap.cc:492] T 2b2db9382a894766af65c470df334a82 P 19a78e3744bb4e3cbb12de42595495ac: No bootstrap required, opened a new log
I20260928 14:53:48.568722 29009 ts_tablet_manager.cc:1397] T 2b2db9382a894766af65c470df334a82 P 19a78e3744bb4e3cbb12de42595495ac: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 14:53:48.568851 29031 ts_tablet_manager.cc:1939] T 7817aaab3f35455aa9b3bc7c3a3c185f P bf2ace9ae64b49d19d11dd8b759ded4e: Deleting consensus metadata
I20260928 14:53:48.568918 29009 raft_consensus.cc:348] T 2b2db9382a894766af65c470df334a82 P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.569000 29009 raft_consensus.cc:374] T 2b2db9382a894766af65c470df334a82 P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:48.569027 29009 raft_consensus.cc:729] T 2b2db9382a894766af65c470df334a82 P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a78e3744bb4e3cbb12de42595495ac, State: Initialized, Role: FOLLOWER
I20260928 14:53:48.569070 29009 consensus_queue.cc:260] T 2b2db9382a894766af65c470df334a82 P 19a78e3744bb4e3cbb12de42595495ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.569169 29009 ts_tablet_manager.cc:1428] T 2b2db9382a894766af65c470df334a82 P 19a78e3744bb4e3cbb12de42595495ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:48.569401 28482 catalog_manager.cc:4841] TS bf2ace9ae64b49d19d11dd8b759ded4e (127.20.116.129:34741): tablet 7817aaab3f35455aa9b3bc7c3a3c185f (table test-table [id=281d4c9ab68e4dcdb3b3ae6ff2667d77]) successfully deleted
W20260928 14:53:48.631273 28982 tablet.cc:2369] T 2b2db9382a894766af65c470df334a82 P 19a78e3744bb4e3cbb12de42595495ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:53:48.669339 29016 raft_consensus.cc:482] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:53:48.669454 29016 raft_consensus.cc:504] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.669819 29016 leader_election.cc:288] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf2ace9ae64b49d19d11dd8b759ded4e (127.20.116.129:34741), 19a78e3744bb4e3cbb12de42595495ac (127.20.116.131:36181)
I20260928 14:53:48.673223 29006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.670059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:40617 (local address 127.20.116.129:34741)
0928 14:53:48.670101 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.670103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.670113 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:53:48.670232 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.670233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.670244 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.670291 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:48.670505 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.671282 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.672508 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.672846 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.672929 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.672939 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:48.673106 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 14:53:48.673143 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:53:48.673223 28734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.670007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:34741 (local address 127.20.116.130:40617)
0928 14:53:48.670067 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:48.670079 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:48.670130 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:48.670340 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:48.670341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:48.670342 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:48.670413 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.670414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.671361 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.671361 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.672377 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.672380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.672927 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.672928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.673019 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.673023 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:53:48.673097 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 14:53:48.673143 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 14:53:48.673473 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.670232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:32877 (local address 127.20.116.131:36181)
0928 14:53:48.670278 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.670280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.670289 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:53:48.670316 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.670317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.670328 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.670390 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:53:48.670568 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.671032 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.672471 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.673063 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.673139 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.673149 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:48.673362 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 14:53:48.673403 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:53:48.673523 28655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2db9382a894766af65c470df334a82" candidate_uuid: "bfdbd4a7b6004d4db726b89031018cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" is_pre_election: true
I20260928 14:53:48.673609 28655 raft_consensus.cc:2454] T 2b2db9382a894766af65c470df334a82 P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfdbd4a7b6004d4db726b89031018cb8 in term 0.
I20260928 14:53:48.673763 28726 leader_election.cc:302] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf2ace9ae64b49d19d11dd8b759ded4e, bfdbd4a7b6004d4db726b89031018cb8; no voters:
I20260928 14:53:48.673887 29037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.670011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:36181 (local address 127.20.116.130:32877)
0928 14:53:48.670160 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:48.670169 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:48.670214 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:48.670400 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:48.670400 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:48.670402 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:53:48.670472 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.670474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.671119 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.671120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.672377 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.672380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.673206 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.673207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.673277 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.673282 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:53:48.673599 (+ 317us) client_negotiation.cc:240] Negotiation successful
0928 14:53:48.673655 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":19,"thread_start_us":69,"threads_started":1}
I20260928 14:53:48.673916 29016 raft_consensus.cc:2781] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:53:48.673952 29016 raft_consensus.cc:482] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:53:48.673977 29016 raft_consensus.cc:3037] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:48.674170 28935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2db9382a894766af65c470df334a82" candidate_uuid: "bfdbd4a7b6004d4db726b89031018cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a78e3744bb4e3cbb12de42595495ac" is_pre_election: true
I20260928 14:53:48.674299 28935 raft_consensus.cc:2454] T 2b2db9382a894766af65c470df334a82 P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfdbd4a7b6004d4db726b89031018cb8 in term 0.
I20260928 14:53:48.674889 29016 raft_consensus.cc:504] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.675033 29016 leader_election.cc:288] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8 [CANDIDATE]: Term 1 election: Requested vote from peers bf2ace9ae64b49d19d11dd8b759ded4e (127.20.116.129:34741), 19a78e3744bb4e3cbb12de42595495ac (127.20.116.131:36181)
I20260928 14:53:48.675187 28935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2db9382a894766af65c470df334a82" candidate_uuid: "bfdbd4a7b6004d4db726b89031018cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a78e3744bb4e3cbb12de42595495ac"
I20260928 14:53:48.675228 28655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2db9382a894766af65c470df334a82" candidate_uuid: "bfdbd4a7b6004d4db726b89031018cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e"
I20260928 14:53:48.675292 28935 raft_consensus.cc:3037] T 2b2db9382a894766af65c470df334a82 P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:48.675305 28655 raft_consensus.cc:3037] T 2b2db9382a894766af65c470df334a82 P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:48.675874 28935 raft_consensus.cc:2454] T 2b2db9382a894766af65c470df334a82 P 19a78e3744bb4e3cbb12de42595495ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfdbd4a7b6004d4db726b89031018cb8 in term 1.
I20260928 14:53:48.676049 28729 leader_election.cc:302] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19a78e3744bb4e3cbb12de42595495ac, bfdbd4a7b6004d4db726b89031018cb8; no voters:
I20260928 14:53:48.676057 28655 raft_consensus.cc:2454] T 2b2db9382a894766af65c470df334a82 P bf2ace9ae64b49d19d11dd8b759ded4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfdbd4a7b6004d4db726b89031018cb8 in term 1.
I20260928 14:53:48.676203 29016 raft_consensus.cc:2781] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:48.676381 29016 raft_consensus.cc:686] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8 [term 1 LEADER]: Becoming Leader. State: Replica: bfdbd4a7b6004d4db726b89031018cb8, State: Running, Role: LEADER
I20260928 14:53:48.676460 29016 consensus_queue.cc:237] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.677265 28497 catalog_manager.cc:5495] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bfdbd4a7b6004d4db726b89031018cb8 (127.20.116.130). New cstate: current_term: 1 leader_uuid: "bfdbd4a7b6004d4db726b89031018cb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } health_report { overall_health: UNKNOWN } } }
Sep 28 14:53:48 dist-test-slave-pxl6 krb5kdc[28453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607227, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.190@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM
I20260928 14:53:48.710114 29045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.703337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:37713 (local address 127.0.0.1:60580)
0928 14:53:48.703500 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:48.703510 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:48.703578 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:48.703731 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:48.703732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:48.704036 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:48.704088 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.704090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.704788 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.704788 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.705323 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.705325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.705704 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.705705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.705748 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.706263 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:48.706268 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:48.707796 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:48.709158 (+ 1362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:48.709165 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:48.709168 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:48.709363 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:48.709563 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:48.709564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:48.709564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:48.709620 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:48.709823 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:48.709829 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:48.709870 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 14:53:48.709921 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 14:53:48.710011 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
I20260928 14:53:48.710129 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.703405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60580 (local address 127.20.116.190:37713)
0928 14:53:48.703493 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.703495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.703577 (+ 82us) server_negotiation.cc:407] Connection header received
0928 14:53:48.703640 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.703640 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.703650 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.703703 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:48.704166 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.704712 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.705401 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.705626 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.705685 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.707900 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:48.707902 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:48.707902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:48.707908 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:48.709086 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:48.709427 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:48.709428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:48.709428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:48.709474 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:48.709674 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:48.709675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:48.709675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:48.709771 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:48.709821 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:48.709953 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 14:53:48.710053 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":60}
I20260928 14:53:48.712321 28497 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:60580:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 14:53:48.712556 28497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:53:48.714286 28635 tablet_service.cc:1467] Processing CreateTablet for tablet 19e05527803d4165b4b5d4a7aac0cb2c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=92b8ff41733e4c4c8af7037659de8b25]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:53:48.714277 28915 tablet_service.cc:1467] Processing CreateTablet for tablet 19e05527803d4165b4b5d4a7aac0cb2c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=92b8ff41733e4c4c8af7037659de8b25]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:53:48.714282 28775 tablet_service.cc:1467] Processing CreateTablet for tablet 19e05527803d4165b4b5d4a7aac0cb2c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=92b8ff41733e4c4c8af7037659de8b25]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:53:48.714437 28915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19e05527803d4165b4b5d4a7aac0cb2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:48.714448 28635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19e05527803d4165b4b5d4a7aac0cb2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:48.714453 28775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19e05527803d4165b4b5d4a7aac0cb2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:48.715903 29014 tablet_bootstrap.cc:492] T 19e05527803d4165b4b5d4a7aac0cb2c P bf2ace9ae64b49d19d11dd8b759ded4e: Bootstrap starting.
I20260928 14:53:48.715988 29010 tablet_bootstrap.cc:492] T 19e05527803d4165b4b5d4a7aac0cb2c P bfdbd4a7b6004d4db726b89031018cb8: Bootstrap starting.
I20260928 14:53:48.716354 29010 tablet_bootstrap.cc:654] T 19e05527803d4165b4b5d4a7aac0cb2c P bfdbd4a7b6004d4db726b89031018cb8: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:48.716352 29014 tablet_bootstrap.cc:654] T 19e05527803d4165b4b5d4a7aac0cb2c P bf2ace9ae64b49d19d11dd8b759ded4e: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:48.716725 29009 tablet_bootstrap.cc:492] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac: Bootstrap starting.
I20260928 14:53:48.717052 29009 tablet_bootstrap.cc:654] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:48.717078 29010 tablet_bootstrap.cc:492] T 19e05527803d4165b4b5d4a7aac0cb2c P bfdbd4a7b6004d4db726b89031018cb8: No bootstrap required, opened a new log
I20260928 14:53:48.717123 29010 ts_tablet_manager.cc:1397] T 19e05527803d4165b4b5d4a7aac0cb2c P bfdbd4a7b6004d4db726b89031018cb8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:48.717234 29014 tablet_bootstrap.cc:492] T 19e05527803d4165b4b5d4a7aac0cb2c P bf2ace9ae64b49d19d11dd8b759ded4e: No bootstrap required, opened a new log
I20260928 14:53:48.717270 29014 ts_tablet_manager.cc:1397] T 19e05527803d4165b4b5d4a7aac0cb2c P bf2ace9ae64b49d19d11dd8b759ded4e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:48.717269 29010 raft_consensus.cc:348] T 19e05527803d4165b4b5d4a7aac0cb2c P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.717348 29010 raft_consensus.cc:374] T 19e05527803d4165b4b5d4a7aac0cb2c P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:48.717370 29010 raft_consensus.cc:729] T 19e05527803d4165b4b5d4a7aac0cb2c P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfdbd4a7b6004d4db726b89031018cb8, State: Initialized, Role: FOLLOWER
I20260928 14:53:48.717417 29010 consensus_queue.cc:260] T 19e05527803d4165b4b5d4a7aac0cb2c P bfdbd4a7b6004d4db726b89031018cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.717403 29014 raft_consensus.cc:348] T 19e05527803d4165b4b5d4a7aac0cb2c P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.717468 29014 raft_consensus.cc:374] T 19e05527803d4165b4b5d4a7aac0cb2c P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:48.717489 29014 raft_consensus.cc:729] T 19e05527803d4165b4b5d4a7aac0cb2c P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2ace9ae64b49d19d11dd8b759ded4e, State: Initialized, Role: FOLLOWER
I20260928 14:53:48.717516 29010 ts_tablet_manager.cc:1428] T 19e05527803d4165b4b5d4a7aac0cb2c P bfdbd4a7b6004d4db726b89031018cb8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:48.717533 29014 consensus_queue.cc:260] T 19e05527803d4165b4b5d4a7aac0cb2c P bf2ace9ae64b49d19d11dd8b759ded4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.717635 29014 ts_tablet_manager.cc:1428] T 19e05527803d4165b4b5d4a7aac0cb2c P bf2ace9ae64b49d19d11dd8b759ded4e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:48.717638 29016 tablet_replica.cc:439] T 19e05527803d4165b4b5d4a7aac0cb2c P bfdbd4a7b6004d4db726b89031018cb8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } } }
I20260928 14:53:48.717716 29016 tablet_replica.cc:442] T 19e05527803d4165b4b5d4a7aac0cb2c P bfdbd4a7b6004d4db726b89031018cb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:53:48.717690 29017 tablet_replica.cc:439] T 19e05527803d4165b4b5d4a7aac0cb2c P bf2ace9ae64b49d19d11dd8b759ded4e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } } }
I20260928 14:53:48.717777 29017 tablet_replica.cc:442] T 19e05527803d4165b4b5d4a7aac0cb2c P bf2ace9ae64b49d19d11dd8b759ded4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:53:48.718312 29009 tablet_bootstrap.cc:492] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac: No bootstrap required, opened a new log
I20260928 14:53:48.718353 29009 ts_tablet_manager.cc:1397] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:53:48.718480 29009 raft_consensus.cc:348] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.718591 29009 raft_consensus.cc:374] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:48.718622 29009 raft_consensus.cc:729] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a78e3744bb4e3cbb12de42595495ac, State: Initialized, Role: FOLLOWER
I20260928 14:53:48.718674 29009 consensus_queue.cc:260] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.718778 29009 ts_tablet_manager.cc:1428] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:48.718804 29019 tablet_replica.cc:439] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } } }
I20260928 14:53:48.718879 29019 tablet_replica.cc:442] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:53:48.835147 29019 raft_consensus.cc:482] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:53:48.835254 29019 raft_consensus.cc:504] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.835453 29019 leader_election.cc:288] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf2ace9ae64b49d19d11dd8b759ded4e (127.20.116.129:34741), bfdbd4a7b6004d4db726b89031018cb8 (127.20.116.130:45141)
I20260928 14:53:48.835723 28655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19e05527803d4165b4b5d4a7aac0cb2c" candidate_uuid: "19a78e3744bb4e3cbb12de42595495ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" is_pre_election: true
I20260928 14:53:48.835821 28655 raft_consensus.cc:2454] T 19e05527803d4165b4b5d4a7aac0cb2c P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19a78e3744bb4e3cbb12de42595495ac in term 0.
I20260928 14:53:48.835762 28795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19e05527803d4165b4b5d4a7aac0cb2c" candidate_uuid: "19a78e3744bb4e3cbb12de42595495ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfdbd4a7b6004d4db726b89031018cb8" is_pre_election: true
I20260928 14:53:48.835891 28795 raft_consensus.cc:2454] T 19e05527803d4165b4b5d4a7aac0cb2c P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19a78e3744bb4e3cbb12de42595495ac in term 0.
I20260928 14:53:48.836104 28866 leader_election.cc:302] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19a78e3744bb4e3cbb12de42595495ac, bf2ace9ae64b49d19d11dd8b759ded4e; no voters:
I20260928 14:53:48.836242 29019 raft_consensus.cc:2781] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:53:48.836298 29019 raft_consensus.cc:482] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:53:48.836338 29019 raft_consensus.cc:3037] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:48.837095 29019 raft_consensus.cc:504] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.837245 29019 leader_election.cc:288] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac [CANDIDATE]: Term 1 election: Requested vote from peers bf2ace9ae64b49d19d11dd8b759ded4e (127.20.116.129:34741), bfdbd4a7b6004d4db726b89031018cb8 (127.20.116.130:45141)
I20260928 14:53:48.837414 28655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19e05527803d4165b4b5d4a7aac0cb2c" candidate_uuid: "19a78e3744bb4e3cbb12de42595495ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e"
I20260928 14:53:48.837416 28795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19e05527803d4165b4b5d4a7aac0cb2c" candidate_uuid: "19a78e3744bb4e3cbb12de42595495ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfdbd4a7b6004d4db726b89031018cb8"
I20260928 14:53:48.837481 28795 raft_consensus.cc:3037] T 19e05527803d4165b4b5d4a7aac0cb2c P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:48.837503 28655 raft_consensus.cc:3037] T 19e05527803d4165b4b5d4a7aac0cb2c P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:48.838101 28655 raft_consensus.cc:2454] T 19e05527803d4165b4b5d4a7aac0cb2c P bf2ace9ae64b49d19d11dd8b759ded4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19a78e3744bb4e3cbb12de42595495ac in term 1.
I20260928 14:53:48.838104 28795 raft_consensus.cc:2454] T 19e05527803d4165b4b5d4a7aac0cb2c P bfdbd4a7b6004d4db726b89031018cb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19a78e3744bb4e3cbb12de42595495ac in term 1.
I20260928 14:53:48.838308 28869 leader_election.cc:302] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19a78e3744bb4e3cbb12de42595495ac, bfdbd4a7b6004d4db726b89031018cb8; no voters:
I20260928 14:53:48.838402 29019 raft_consensus.cc:2781] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:48.838446 29019 raft_consensus.cc:686] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac [term 1 LEADER]: Becoming Leader. State: Replica: 19a78e3744bb4e3cbb12de42595495ac, State: Running, Role: LEADER
I20260928 14:53:48.838500 29019 consensus_queue.cc:237] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.838809 29013 tablet_replica.cc:439] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac: TxnStatusTablet state changed. Reason: New leader 19a78e3744bb4e3cbb12de42595495ac. Latest consensus state: current_term: 1 leader_uuid: "19a78e3744bb4e3cbb12de42595495ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } } }
I20260928 14:53:48.838904 29013 tablet_replica.cc:442] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:53:48.839123 29050 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 14:53:48.839270 29050 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 14:53:48.839201 28498 catalog_manager.cc:5495] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 19a78e3744bb4e3cbb12de42595495ac (127.20.116.131). New cstate: current_term: 1 leader_uuid: "19a78e3744bb4e3cbb12de42595495ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } health_report { overall_health: HEALTHY } } }
I20260928 14:53:48.866773 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.864314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44308 (local address 127.20.116.131:36181)
0928 14:53:48.864391 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.864394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.864404 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:53:48.864496 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.864497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.864509 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.864560 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:48.864704 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.865189 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.865994 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.866209 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.866286 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.866474 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:48.866513 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:48.866595 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:48.866664 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 14:53:48.866689 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:53:48.866782 29045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.864245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:36181 (local address 127.0.0.1:44308)
0928 14:53:48.864353 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:48.864372 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:48.864427 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:48.864565 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:48.864566 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:48.864567 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:53:48.864630 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.864632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.865278 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.865280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.865900 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.865902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.866287 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.866288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.866366 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.866384 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:53:48.866597 (+ 213us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:53:48.866615 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 14:53:48.866662 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 14:53:48.866712 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 14:53:48.867906 28795 raft_consensus.cc:1261] T 19e05527803d4165b4b5d4a7aac0cb2c P bfdbd4a7b6004d4db726b89031018cb8 [term 1 FOLLOWER]: Refusing update from remote peer 19a78e3744bb4e3cbb12de42595495ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:48.867909 28655 raft_consensus.cc:1261] T 19e05527803d4165b4b5d4a7aac0cb2c P bf2ace9ae64b49d19d11dd8b759ded4e [term 1 FOLLOWER]: Refusing update from remote peer 19a78e3744bb4e3cbb12de42595495ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:48.868144 29019 consensus_queue.cc:1035] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:48.868258 29013 consensus_queue.cc:1035] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:48.868767 29016 tablet_replica.cc:439] T 19e05527803d4165b4b5d4a7aac0cb2c P bfdbd4a7b6004d4db726b89031018cb8: TxnStatusTablet state changed. Reason: New leader 19a78e3744bb4e3cbb12de42595495ac. Latest consensus state: current_term: 1 leader_uuid: "19a78e3744bb4e3cbb12de42595495ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } } }
I20260928 14:53:48.868861 29016 tablet_replica.cc:442] T 19e05527803d4165b4b5d4a7aac0cb2c P bfdbd4a7b6004d4db726b89031018cb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:53:48.869036 29017 tablet_replica.cc:439] T 19e05527803d4165b4b5d4a7aac0cb2c P bf2ace9ae64b49d19d11dd8b759ded4e: TxnStatusTablet state changed. Reason: New leader 19a78e3744bb4e3cbb12de42595495ac. Latest consensus state: current_term: 1 leader_uuid: "19a78e3744bb4e3cbb12de42595495ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } } }
I20260928 14:53:48.869128 29017 tablet_replica.cc:442] T 19e05527803d4165b4b5d4a7aac0cb2c P bf2ace9ae64b49d19d11dd8b759ded4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:53:48.869621 29013 tablet_replica.cc:439] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19a78e3744bb4e3cbb12de42595495ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } } }
I20260928 14:53:48.869706 29013 tablet_replica.cc:442] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:53:48.869769 29022 mvcc.cc:205] Tried to move back new op lower bound from 7334327209441132544 to 7334327209322975232. Current Snapshot: MvccSnapshot[applied={T|T < 7334327209441132544 or (T in {7334327209441132544})}]
I20260928 14:53:48.869913 29023 mvcc.cc:205] Tried to move back new op lower bound from 7334327209441132544 to 7334327209322975232. Current Snapshot: MvccSnapshot[applied={T|T < 7334327209441132544}]
I20260928 14:53:48.869992 29019 tablet_replica.cc:439] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19a78e3744bb4e3cbb12de42595495ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } } }
I20260928 14:53:48.870004 29016 tablet_replica.cc:439] T 19e05527803d4165b4b5d4a7aac0cb2c P bfdbd4a7b6004d4db726b89031018cb8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19a78e3744bb4e3cbb12de42595495ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } } }
I20260928 14:53:48.870069 29019 tablet_replica.cc:442] T 19e05527803d4165b4b5d4a7aac0cb2c P 19a78e3744bb4e3cbb12de42595495ac: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:53:48.870070 29016 tablet_replica.cc:442] T 19e05527803d4165b4b5d4a7aac0cb2c P bfdbd4a7b6004d4db726b89031018cb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:53:48.870993 29017 tablet_replica.cc:439] T 19e05527803d4165b4b5d4a7aac0cb2c P bf2ace9ae64b49d19d11dd8b759ded4e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19a78e3744bb4e3cbb12de42595495ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } } }
I20260928 14:53:48.871070 29017 tablet_replica.cc:442] T 19e05527803d4165b4b5d4a7aac0cb2c P bf2ace9ae64b49d19d11dd8b759ded4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:53:48.873965 29005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.871621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57774 (local address 127.20.116.130:45141)
0928 14:53:48.871658 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.871660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.871670 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:53:48.871706 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.871707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.871726 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.871800 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:48.871960 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.872440 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.873183 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.873394 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.873471 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.873627 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:48.873746 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:48.873827 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:48.873869 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 14:53:48.873889 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:53:48.878937 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.876664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:45385 (local address 127.20.116.131:36181)
0928 14:53:48.876728 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.876730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.876754 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:53:48.876823 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.876824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.876835 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.876889 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:48.877011 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.877459 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.878217 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.878408 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.878482 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.878690 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:48.878725 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:48.878781 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:48.878840 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 14:53:48.878862 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 14:53:48.879149 29058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.876496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:36181 (local address 127.20.116.130:45385)
0928 14:53:48.876693 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:48.876706 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:48.876772 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:48.876893 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:48.876893 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:48.876895 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:53:48.876948 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.876949 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.877546 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.877547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.878117 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.878119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.878500 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.878501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.878585 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.878601 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:53:48.878794 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:53:48.878796 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:53:48.878839 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 14:53:48.878882 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":97,"threads_started":1}
I20260928 14:53:48.881518 28655 raft_consensus.cc:1261] T 2b2db9382a894766af65c470df334a82 P bf2ace9ae64b49d19d11dd8b759ded4e [term 1 FOLLOWER]: Refusing update from remote peer bfdbd4a7b6004d4db726b89031018cb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:48.881681 28935 raft_consensus.cc:1261] T 2b2db9382a894766af65c470df334a82 P 19a78e3744bb4e3cbb12de42595495ac [term 1 FOLLOWER]: Refusing update from remote peer bfdbd4a7b6004d4db726b89031018cb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:48.881800 29016 consensus_queue.cc:1035] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:48.881947 29016 consensus_queue.cc:1035] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:48.890719 29005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.888069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:42981 (local address 127.20.116.130:45141)
0928 14:53:48.888108 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.888111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.888122 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:53:48.888144 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.888145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.888161 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.888207 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:48.888349 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.888813 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.889930 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.890216 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.890291 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.890453 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:48.890488 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:48.890557 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:48.890623 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 14:53:48.890644 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:53:48.890724 28987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.887980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:45141 (local address 127.20.116.131:42981)
0928 14:53:48.888041 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:48.888053 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:48.888103 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:48.888235 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:48.888235 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:48.888237 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:53:48.888285 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.888287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.888914 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.888915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.889803 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:48.889810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.890290 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.890291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:48.890349 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.890364 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:53:48.890563 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:53:48.890565 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:53:48.890616 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 14:53:48.890660 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 14:53:48.899715 28497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:48.899828 28497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:48.900530 28497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b: Sending DeleteTablet for 3 replicas of tablet 2b2db9382a894766af65c470df334a82
I20260928 14:53:48.900796 28635 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b2db9382a894766af65c470df334a82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:48 UTC) from {username='oryx', principal='oryx/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:43376
I20260928 14:53:48.900853 28915 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b2db9382a894766af65c470df334a82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:48 UTC) from {username='oryx', principal='oryx/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:44284
I20260928 14:53:48.900957 29032 tablet_replica.cc:330] T 2b2db9382a894766af65c470df334a82 P 19a78e3744bb4e3cbb12de42595495ac: stopping tablet replica
I20260928 14:53:48.900964 29031 tablet_replica.cc:330] T 2b2db9382a894766af65c470df334a82 P bf2ace9ae64b49d19d11dd8b759ded4e: stopping tablet replica
I20260928 14:53:48.901015 29032 raft_consensus.cc:2229] T 2b2db9382a894766af65c470df334a82 P 19a78e3744bb4e3cbb12de42595495ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:48.901023 29031 raft_consensus.cc:2229] T 2b2db9382a894766af65c470df334a82 P bf2ace9ae64b49d19d11dd8b759ded4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:48.901050 29032 raft_consensus.cc:2258] T 2b2db9382a894766af65c470df334a82 P 19a78e3744bb4e3cbb12de42595495ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:48.901054 28775 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b2db9382a894766af65c470df334a82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:48 UTC) from {username='oryx', principal='oryx/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:57758
I20260928 14:53:48.901096 29031 raft_consensus.cc:2258] T 2b2db9382a894766af65c470df334a82 P bf2ace9ae64b49d19d11dd8b759ded4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:48.901176 29033 tablet_replica.cc:330] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8: stopping tablet replica
I20260928 14:53:48.901232 29033 raft_consensus.cc:2229] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:53:48.901324 29033 raft_consensus.cc:2258] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:48.901582 29032 ts_tablet_manager.cc:1905] T 2b2db9382a894766af65c470df334a82 P 19a78e3744bb4e3cbb12de42595495ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:48.901742 29033 ts_tablet_manager.cc:1905] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:48.903038 29031 ts_tablet_manager.cc:1905] T 2b2db9382a894766af65c470df334a82 P bf2ace9ae64b49d19d11dd8b759ded4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:48.903352 29033 ts_tablet_manager.cc:1918] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:53:48.903412 29033 log.cc:1196] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/2b2db9382a894766af65c470df334a82
I20260928 14:53:48.903434 29032 ts_tablet_manager.cc:1918] T 2b2db9382a894766af65c470df334a82 P 19a78e3744bb4e3cbb12de42595495ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:53:48.903483 29032 log.cc:1196] T 2b2db9382a894766af65c470df334a82 P 19a78e3744bb4e3cbb12de42595495ac: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/2b2db9382a894766af65c470df334a82
I20260928 14:53:48.903640 29033 ts_tablet_manager.cc:1939] T 2b2db9382a894766af65c470df334a82 P bfdbd4a7b6004d4db726b89031018cb8: Deleting consensus metadata
I20260928 14:53:48.903710 29032 ts_tablet_manager.cc:1939] T 2b2db9382a894766af65c470df334a82 P 19a78e3744bb4e3cbb12de42595495ac: Deleting consensus metadata
I20260928 14:53:48.904083 28485 catalog_manager.cc:4841] TS 19a78e3744bb4e3cbb12de42595495ac (127.20.116.131:36181): tablet 2b2db9382a894766af65c470df334a82 (table test-table [id=a91b0c8245a14495bbc8d91c52a92f9e]) successfully deleted
I20260928 14:53:48.904767 29031 ts_tablet_manager.cc:1918] T 2b2db9382a894766af65c470df334a82 P bf2ace9ae64b49d19d11dd8b759ded4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:53:48.904843 29031 log.cc:1196] T 2b2db9382a894766af65c470df334a82 P bf2ace9ae64b49d19d11dd8b759ded4e: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-0/wal/wals/2b2db9382a894766af65c470df334a82
I20260928 14:53:48.905052 29031 ts_tablet_manager.cc:1939] T 2b2db9382a894766af65c470df334a82 P bf2ace9ae64b49d19d11dd8b759ded4e: Deleting consensus metadata
I20260928 14:53:48.905740 28482 catalog_manager.cc:4841] TS bf2ace9ae64b49d19d11dd8b759ded4e (127.20.116.129:34741): tablet 2b2db9382a894766af65c470df334a82 (table test-table [id=a91b0c8245a14495bbc8d91c52a92f9e]) successfully deleted
I20260928 14:53:48.906829 28485 catalog_manager.cc:4841] TS bfdbd4a7b6004d4db726b89031018cb8 (127.20.116.130:45141): tablet 2b2db9382a894766af65c470df334a82 (table test-table [id=a91b0c8245a14495bbc8d91c52a92f9e]) successfully deleted
Sep 28 14:53:48 dist-test-slave-pxl6 krb5kdc[28453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:53:48 dist-test-slave-pxl6 krb5kdc[28453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:48 dist-test-slave-pxl6 krb5kdc[28453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:53:48.934065 28547 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:48.926085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.20.116.190:37713)
0928 14:53:48.926152 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.926154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.926413 (+ 259us) server_negotiation.cc:407] Connection header received
0928 14:53:48.926463 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.926464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.926480 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.926709 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:48.927550 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.928439 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.929583 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.929942 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.930032 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.933974 (+ 3942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60592: BlockingRecv error: recv got EOF from 127.0.0.1:60592 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 14:53:48 dist-test-slave-pxl6 krb5kdc[28453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM
I20260928 14:53:48.945214 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:48.935630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60596 (local address 127.20.116.190:37713)
0928 14:53:48.935676 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:48.935678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:48.935750 (+ 72us) server_negotiation.cc:407] Connection header received
0928 14:53:48.935872 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:48.935874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:48.935886 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:48.935951 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:48.936713 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.937501 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.938666 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:48.938970 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:48.939054 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:48.941980 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:48.941981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:48.941982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:48.941991 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:48.943965 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:48.944370 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:48.944371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:48.944372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:48.944428 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:48.944623 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:48.944624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:48.944624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:48.944754 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:48.944829 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:48.945033 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 14:53:48.945121 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:53:48.947232 28497 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:53:48.947417 28497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:53:48.949206 28775 tablet_service.cc:1467] Processing CreateTablet for tablet 20e418fed0f14d2c840e602486819463 (DEFAULT_TABLE table=test-table [id=7ca7e20408534a1d89e857a858636d26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:48.949353 28775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e418fed0f14d2c840e602486819463. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:48.949322 28635 tablet_service.cc:1467] Processing CreateTablet for tablet 20e418fed0f14d2c840e602486819463 (DEFAULT_TABLE table=test-table [id=7ca7e20408534a1d89e857a858636d26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:48.949479 28635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e418fed0f14d2c840e602486819463. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:48.950809 29010 tablet_bootstrap.cc:492] T 20e418fed0f14d2c840e602486819463 P bfdbd4a7b6004d4db726b89031018cb8: Bootstrap starting.
I20260928 14:53:48.951082 28915 tablet_service.cc:1467] Processing CreateTablet for tablet 20e418fed0f14d2c840e602486819463 (DEFAULT_TABLE table=test-table [id=7ca7e20408534a1d89e857a858636d26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:48.951210 28915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e418fed0f14d2c840e602486819463. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:48.951253 29010 tablet_bootstrap.cc:654] T 20e418fed0f14d2c840e602486819463 P bfdbd4a7b6004d4db726b89031018cb8: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:48.952595 29009 tablet_bootstrap.cc:492] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac: Bootstrap starting.
I20260928 14:53:48.952709 29014 tablet_bootstrap.cc:492] T 20e418fed0f14d2c840e602486819463 P bf2ace9ae64b49d19d11dd8b759ded4e: Bootstrap starting.
I20260928 14:53:48.952955 29009 tablet_bootstrap.cc:654] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:48.953140 29014 tablet_bootstrap.cc:654] T 20e418fed0f14d2c840e602486819463 P bf2ace9ae64b49d19d11dd8b759ded4e: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:48.954109 29014 tablet_bootstrap.cc:492] T 20e418fed0f14d2c840e602486819463 P bf2ace9ae64b49d19d11dd8b759ded4e: No bootstrap required, opened a new log
I20260928 14:53:48.954182 29014 ts_tablet_manager.cc:1397] T 20e418fed0f14d2c840e602486819463 P bf2ace9ae64b49d19d11dd8b759ded4e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:48.954350 29014 raft_consensus.cc:348] T 20e418fed0f14d2c840e602486819463 P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.954442 29014 raft_consensus.cc:374] T 20e418fed0f14d2c840e602486819463 P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:48.954464 29014 raft_consensus.cc:729] T 20e418fed0f14d2c840e602486819463 P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2ace9ae64b49d19d11dd8b759ded4e, State: Initialized, Role: FOLLOWER
I20260928 14:53:48.954509 29014 consensus_queue.cc:260] T 20e418fed0f14d2c840e602486819463 P bf2ace9ae64b49d19d11dd8b759ded4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.954604 29010 tablet_bootstrap.cc:492] T 20e418fed0f14d2c840e602486819463 P bfdbd4a7b6004d4db726b89031018cb8: No bootstrap required, opened a new log
I20260928 14:53:48.954627 29014 ts_tablet_manager.cc:1428] T 20e418fed0f14d2c840e602486819463 P bf2ace9ae64b49d19d11dd8b759ded4e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:53:48.954658 29010 ts_tablet_manager.cc:1397] T 20e418fed0f14d2c840e602486819463 P bfdbd4a7b6004d4db726b89031018cb8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 14:53:48.954820 29010 raft_consensus.cc:348] T 20e418fed0f14d2c840e602486819463 P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.954888 29010 raft_consensus.cc:374] T 20e418fed0f14d2c840e602486819463 P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:48.954908 29010 raft_consensus.cc:729] T 20e418fed0f14d2c840e602486819463 P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfdbd4a7b6004d4db726b89031018cb8, State: Initialized, Role: FOLLOWER
I20260928 14:53:48.954959 29010 consensus_queue.cc:260] T 20e418fed0f14d2c840e602486819463 P bfdbd4a7b6004d4db726b89031018cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.955050 29010 ts_tablet_manager.cc:1428] T 20e418fed0f14d2c840e602486819463 P bfdbd4a7b6004d4db726b89031018cb8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:48.959448 29009 tablet_bootstrap.cc:492] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac: No bootstrap required, opened a new log
I20260928 14:53:48.959525 29009 ts_tablet_manager.cc:1397] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260928 14:53:48.959729 29009 raft_consensus.cc:348] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.959833 29009 raft_consensus.cc:374] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:48.959856 29009 raft_consensus.cc:729] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a78e3744bb4e3cbb12de42595495ac, State: Initialized, Role: FOLLOWER
I20260928 14:53:48.959908 29009 consensus_queue.cc:260] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:48.960002 29009 ts_tablet_manager.cc:1428] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:53:49.177235 29013 raft_consensus.cc:482] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:53:49.177343 29013 raft_consensus.cc:504] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:49.177546 29013 leader_election.cc:288] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf2ace9ae64b49d19d11dd8b759ded4e (127.20.116.129:34741), bfdbd4a7b6004d4db726b89031018cb8 (127.20.116.130:45141)
I20260928 14:53:49.177940 28655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e418fed0f14d2c840e602486819463" candidate_uuid: "19a78e3744bb4e3cbb12de42595495ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" is_pre_election: true
I20260928 14:53:49.177955 28795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e418fed0f14d2c840e602486819463" candidate_uuid: "19a78e3744bb4e3cbb12de42595495ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfdbd4a7b6004d4db726b89031018cb8" is_pre_election: true
I20260928 14:53:49.178087 28795 raft_consensus.cc:2454] T 20e418fed0f14d2c840e602486819463 P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19a78e3744bb4e3cbb12de42595495ac in term 0.
I20260928 14:53:49.178090 28655 raft_consensus.cc:2454] T 20e418fed0f14d2c840e602486819463 P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19a78e3744bb4e3cbb12de42595495ac in term 0.
I20260928 14:53:49.178308 28869 leader_election.cc:302] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19a78e3744bb4e3cbb12de42595495ac, bfdbd4a7b6004d4db726b89031018cb8; no voters:
I20260928 14:53:49.178402 29013 raft_consensus.cc:2781] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:53:49.178447 29013 raft_consensus.cc:482] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:53:49.178467 29013 raft_consensus.cc:3037] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:49.179234 29013 raft_consensus.cc:504] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:49.179376 29013 leader_election.cc:288] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac [CANDIDATE]: Term 1 election: Requested vote from peers bf2ace9ae64b49d19d11dd8b759ded4e (127.20.116.129:34741), bfdbd4a7b6004d4db726b89031018cb8 (127.20.116.130:45141)
I20260928 14:53:49.179561 28655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e418fed0f14d2c840e602486819463" candidate_uuid: "19a78e3744bb4e3cbb12de42595495ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e"
I20260928 14:53:49.179628 28655 raft_consensus.cc:3037] T 20e418fed0f14d2c840e602486819463 P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:49.179561 28795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e418fed0f14d2c840e602486819463" candidate_uuid: "19a78e3744bb4e3cbb12de42595495ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfdbd4a7b6004d4db726b89031018cb8"
I20260928 14:53:49.179661 28795 raft_consensus.cc:3037] T 20e418fed0f14d2c840e602486819463 P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:49.180238 28655 raft_consensus.cc:2454] T 20e418fed0f14d2c840e602486819463 P bf2ace9ae64b49d19d11dd8b759ded4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19a78e3744bb4e3cbb12de42595495ac in term 1.
I20260928 14:53:49.180346 28795 raft_consensus.cc:2454] T 20e418fed0f14d2c840e602486819463 P bfdbd4a7b6004d4db726b89031018cb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19a78e3744bb4e3cbb12de42595495ac in term 1.
I20260928 14:53:49.180414 28866 leader_election.cc:302] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19a78e3744bb4e3cbb12de42595495ac, bf2ace9ae64b49d19d11dd8b759ded4e; no voters:
I20260928 14:53:49.180521 29013 raft_consensus.cc:2781] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:49.180569 29013 raft_consensus.cc:686] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac [term 1 LEADER]: Becoming Leader. State: Replica: 19a78e3744bb4e3cbb12de42595495ac, State: Running, Role: LEADER
I20260928 14:53:49.180629 29013 consensus_queue.cc:237] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:49.181270 28497 catalog_manager.cc:5495] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 19a78e3744bb4e3cbb12de42595495ac (127.20.116.131). New cstate: current_term: 1 leader_uuid: "19a78e3744bb4e3cbb12de42595495ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } health_report { overall_health: HEALTHY } } }
I20260928 14:53:49.238631 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:49.236341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44312 (local address 127.20.116.131:36181)
0928 14:53:49.236421 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:49.236423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:49.236432 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:53:49.236463 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:49.236464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:49.236476 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:49.236536 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:49.236683 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:49.237160 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:49.237914 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:49.238134 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:49.238215 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:49.238359 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:49.238395 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:49.238444 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:49.238493 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 14:53:49.238514 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:53:49.239507 28655 raft_consensus.cc:1261] T 20e418fed0f14d2c840e602486819463 P bf2ace9ae64b49d19d11dd8b759ded4e [term 1 FOLLOWER]: Refusing update from remote peer 19a78e3744bb4e3cbb12de42595495ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:49.239513 28795 raft_consensus.cc:1261] T 20e418fed0f14d2c840e602486819463 P bfdbd4a7b6004d4db726b89031018cb8 [term 1 FOLLOWER]: Refusing update from remote peer 19a78e3744bb4e3cbb12de42595495ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:49.239777 29055 consensus_queue.cc:1035] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:49.239876 29013 consensus_queue.cc:1035] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:49.242197 28498 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:49.242290 28498 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:49.242901 28498 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b: Sending DeleteTablet for 3 replicas of tablet 20e418fed0f14d2c840e602486819463
I20260928 14:53:49.243187 28635 tablet_service.cc:1514] Processing DeleteTablet for tablet 20e418fed0f14d2c840e602486819463 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:49 UTC) from {username='oryx', principal='oryx/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:43376
I20260928 14:53:49.243216 28775 tablet_service.cc:1514] Processing DeleteTablet for tablet 20e418fed0f14d2c840e602486819463 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:49 UTC) from {username='oryx', principal='oryx/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:57758
I20260928 14:53:49.243283 28915 tablet_service.cc:1514] Processing DeleteTablet for tablet 20e418fed0f14d2c840e602486819463 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:49 UTC) from {username='oryx', principal='oryx/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:44284
I20260928 14:53:49.243309 29031 tablet_replica.cc:330] T 20e418fed0f14d2c840e602486819463 P bf2ace9ae64b49d19d11dd8b759ded4e: stopping tablet replica
I20260928 14:53:49.243366 29031 raft_consensus.cc:2229] T 20e418fed0f14d2c840e602486819463 P bf2ace9ae64b49d19d11dd8b759ded4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:49.243382 29033 tablet_replica.cc:330] T 20e418fed0f14d2c840e602486819463 P bfdbd4a7b6004d4db726b89031018cb8: stopping tablet replica
I20260928 14:53:49.243417 29031 raft_consensus.cc:2258] T 20e418fed0f14d2c840e602486819463 P bf2ace9ae64b49d19d11dd8b759ded4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:49.243439 29033 raft_consensus.cc:2229] T 20e418fed0f14d2c840e602486819463 P bfdbd4a7b6004d4db726b89031018cb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:49.243490 29033 raft_consensus.cc:2258] T 20e418fed0f14d2c840e602486819463 P bfdbd4a7b6004d4db726b89031018cb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:49.243566 29032 tablet_replica.cc:330] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac: stopping tablet replica
I20260928 14:53:49.243629 29032 raft_consensus.cc:2229] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:53:49.243697 29032 raft_consensus.cc:2258] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:49.243762 29031 ts_tablet_manager.cc:1905] T 20e418fed0f14d2c840e602486819463 P bf2ace9ae64b49d19d11dd8b759ded4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:49.243942 28498 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:53:49.244088 28498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:53:49.244445 29033 ts_tablet_manager.cc:1905] T 20e418fed0f14d2c840e602486819463 P bfdbd4a7b6004d4db726b89031018cb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:49.245462 29032 ts_tablet_manager.cc:1905] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:49.245464 29031 ts_tablet_manager.cc:1918] T 20e418fed0f14d2c840e602486819463 P bf2ace9ae64b49d19d11dd8b759ded4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:53:49.245524 29031 log.cc:1196] T 20e418fed0f14d2c840e602486819463 P bf2ace9ae64b49d19d11dd8b759ded4e: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-0/wal/wals/20e418fed0f14d2c840e602486819463
I20260928 14:53:49.245733 29031 ts_tablet_manager.cc:1939] T 20e418fed0f14d2c840e602486819463 P bf2ace9ae64b49d19d11dd8b759ded4e: Deleting consensus metadata
I20260928 14:53:49.245795 29033 ts_tablet_manager.cc:1918] T 20e418fed0f14d2c840e602486819463 P bfdbd4a7b6004d4db726b89031018cb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:53:49.245848 29033 log.cc:1196] T 20e418fed0f14d2c840e602486819463 P bfdbd4a7b6004d4db726b89031018cb8: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/20e418fed0f14d2c840e602486819463
I20260928 14:53:49.246065 29033 ts_tablet_manager.cc:1939] T 20e418fed0f14d2c840e602486819463 P bfdbd4a7b6004d4db726b89031018cb8: Deleting consensus metadata
I20260928 14:53:49.246065 28482 catalog_manager.cc:4841] TS bf2ace9ae64b49d19d11dd8b759ded4e (127.20.116.129:34741): tablet 20e418fed0f14d2c840e602486819463 (table test-table [id=7ca7e20408534a1d89e857a858636d26]) successfully deleted
I20260928 14:53:49.246608 28485 catalog_manager.cc:4841] TS bfdbd4a7b6004d4db726b89031018cb8 (127.20.116.130:45141): tablet 20e418fed0f14d2c840e602486819463 (table test-table [id=7ca7e20408534a1d89e857a858636d26]) successfully deleted
I20260928 14:53:49.246732 28635 tablet_service.cc:1467] Processing CreateTablet for tablet 9a9ae442e48345aa85ba9b00957c96e0 (DEFAULT_TABLE table=test-table [id=73d57a7ed542482db0708ebc85e7c739]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:49.246843 28635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a9ae442e48345aa85ba9b00957c96e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:49.246922 28775 tablet_service.cc:1467] Processing CreateTablet for tablet 9a9ae442e48345aa85ba9b00957c96e0 (DEFAULT_TABLE table=test-table [id=73d57a7ed542482db0708ebc85e7c739]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:49.246963 29032 ts_tablet_manager.cc:1918] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:53:49.247008 29032 log.cc:1196] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/20e418fed0f14d2c840e602486819463
I20260928 14:53:49.247051 28775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a9ae442e48345aa85ba9b00957c96e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:49.247215 29032 ts_tablet_manager.cc:1939] T 20e418fed0f14d2c840e602486819463 P 19a78e3744bb4e3cbb12de42595495ac: Deleting consensus metadata
I20260928 14:53:49.247362 28915 tablet_service.cc:1467] Processing CreateTablet for tablet 9a9ae442e48345aa85ba9b00957c96e0 (DEFAULT_TABLE table=test-table [id=73d57a7ed542482db0708ebc85e7c739]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:53:49.247457 28915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a9ae442e48345aa85ba9b00957c96e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:49.248059 29010 tablet_bootstrap.cc:492] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8: Bootstrap starting.
I20260928 14:53:49.248080 29014 tablet_bootstrap.cc:492] T 9a9ae442e48345aa85ba9b00957c96e0 P bf2ace9ae64b49d19d11dd8b759ded4e: Bootstrap starting.
I20260928 14:53:49.248373 29010 tablet_bootstrap.cc:654] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:49.248677 29014 tablet_bootstrap.cc:654] T 9a9ae442e48345aa85ba9b00957c96e0 P bf2ace9ae64b49d19d11dd8b759ded4e: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:49.248692 29009 tablet_bootstrap.cc:492] T 9a9ae442e48345aa85ba9b00957c96e0 P 19a78e3744bb4e3cbb12de42595495ac: Bootstrap starting.
I20260928 14:53:49.248883 28485 catalog_manager.cc:4841] TS 19a78e3744bb4e3cbb12de42595495ac (127.20.116.131:36181): tablet 20e418fed0f14d2c840e602486819463 (table test-table [id=7ca7e20408534a1d89e857a858636d26]) successfully deleted
I20260928 14:53:49.249030 29009 tablet_bootstrap.cc:654] T 9a9ae442e48345aa85ba9b00957c96e0 P 19a78e3744bb4e3cbb12de42595495ac: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:49.249356 29010 tablet_bootstrap.cc:492] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8: No bootstrap required, opened a new log
I20260928 14:53:49.249413 29010 ts_tablet_manager.cc:1397] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:53:49.249526 29010 raft_consensus.cc:348] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:49.249564 29009 tablet_bootstrap.cc:492] T 9a9ae442e48345aa85ba9b00957c96e0 P 19a78e3744bb4e3cbb12de42595495ac: No bootstrap required, opened a new log
I20260928 14:53:49.249575 29010 raft_consensus.cc:374] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:49.249589 29010 raft_consensus.cc:729] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfdbd4a7b6004d4db726b89031018cb8, State: Initialized, Role: FOLLOWER
I20260928 14:53:49.249598 29009 ts_tablet_manager.cc:1397] T 9a9ae442e48345aa85ba9b00957c96e0 P 19a78e3744bb4e3cbb12de42595495ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:49.249625 29010 consensus_queue.cc:260] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:49.249725 29010 ts_tablet_manager.cc:1428] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:49.249743 29009 raft_consensus.cc:348] T 9a9ae442e48345aa85ba9b00957c96e0 P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:49.249805 29009 raft_consensus.cc:374] T 9a9ae442e48345aa85ba9b00957c96e0 P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:49.249828 29009 raft_consensus.cc:729] T 9a9ae442e48345aa85ba9b00957c96e0 P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a78e3744bb4e3cbb12de42595495ac, State: Initialized, Role: FOLLOWER
I20260928 14:53:49.249878 29009 consensus_queue.cc:260] T 9a9ae442e48345aa85ba9b00957c96e0 P 19a78e3744bb4e3cbb12de42595495ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:49.249984 29009 ts_tablet_manager.cc:1428] T 9a9ae442e48345aa85ba9b00957c96e0 P 19a78e3744bb4e3cbb12de42595495ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:49.250192 29014 tablet_bootstrap.cc:492] T 9a9ae442e48345aa85ba9b00957c96e0 P bf2ace9ae64b49d19d11dd8b759ded4e: No bootstrap required, opened a new log
I20260928 14:53:49.250234 29014 ts_tablet_manager.cc:1397] T 9a9ae442e48345aa85ba9b00957c96e0 P bf2ace9ae64b49d19d11dd8b759ded4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:53:49.250398 29014 raft_consensus.cc:348] T 9a9ae442e48345aa85ba9b00957c96e0 P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:49.250473 29014 raft_consensus.cc:374] T 9a9ae442e48345aa85ba9b00957c96e0 P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:49.250491 29014 raft_consensus.cc:729] T 9a9ae442e48345aa85ba9b00957c96e0 P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2ace9ae64b49d19d11dd8b759ded4e, State: Initialized, Role: FOLLOWER
I20260928 14:53:49.250559 29014 consensus_queue.cc:260] T 9a9ae442e48345aa85ba9b00957c96e0 P bf2ace9ae64b49d19d11dd8b759ded4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:49.250674 29014 ts_tablet_manager.cc:1428] T 9a9ae442e48345aa85ba9b00957c96e0 P bf2ace9ae64b49d19d11dd8b759ded4e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 14:53:49.316277 29061 raft_consensus.cc:482] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:53:49.316390 29061 raft_consensus.cc:504] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:49.316627 29061 leader_election.cc:288] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf2ace9ae64b49d19d11dd8b759ded4e (127.20.116.129:34741), 19a78e3744bb4e3cbb12de42595495ac (127.20.116.131:36181)
I20260928 14:53:49.316951 28935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a9ae442e48345aa85ba9b00957c96e0" candidate_uuid: "bfdbd4a7b6004d4db726b89031018cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a78e3744bb4e3cbb12de42595495ac" is_pre_election: true
I20260928 14:53:49.316954 28655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a9ae442e48345aa85ba9b00957c96e0" candidate_uuid: "bfdbd4a7b6004d4db726b89031018cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" is_pre_election: true
I20260928 14:53:49.317058 28935 raft_consensus.cc:2454] T 9a9ae442e48345aa85ba9b00957c96e0 P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfdbd4a7b6004d4db726b89031018cb8 in term 0.
I20260928 14:53:49.317106 28655 raft_consensus.cc:2454] T 9a9ae442e48345aa85ba9b00957c96e0 P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfdbd4a7b6004d4db726b89031018cb8 in term 0.
I20260928 14:53:49.317242 28729 leader_election.cc:302] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19a78e3744bb4e3cbb12de42595495ac, bfdbd4a7b6004d4db726b89031018cb8; no voters:
I20260928 14:53:49.317337 29061 raft_consensus.cc:2781] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:53:49.317381 29061 raft_consensus.cc:482] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:53:49.317413 29061 raft_consensus.cc:3037] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:49.318143 29061 raft_consensus.cc:504] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:49.318305 29061 leader_election.cc:288] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8 [CANDIDATE]: Term 1 election: Requested vote from peers bf2ace9ae64b49d19d11dd8b759ded4e (127.20.116.129:34741), 19a78e3744bb4e3cbb12de42595495ac (127.20.116.131:36181)
I20260928 14:53:49.318451 28655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a9ae442e48345aa85ba9b00957c96e0" candidate_uuid: "bfdbd4a7b6004d4db726b89031018cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e"
I20260928 14:53:49.318472 28935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a9ae442e48345aa85ba9b00957c96e0" candidate_uuid: "bfdbd4a7b6004d4db726b89031018cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a78e3744bb4e3cbb12de42595495ac"
I20260928 14:53:49.318579 28935 raft_consensus.cc:3037] T 9a9ae442e48345aa85ba9b00957c96e0 P 19a78e3744bb4e3cbb12de42595495ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:49.318586 28655 raft_consensus.cc:3037] T 9a9ae442e48345aa85ba9b00957c96e0 P bf2ace9ae64b49d19d11dd8b759ded4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:49.319162 28935 raft_consensus.cc:2454] T 9a9ae442e48345aa85ba9b00957c96e0 P 19a78e3744bb4e3cbb12de42595495ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfdbd4a7b6004d4db726b89031018cb8 in term 1.
I20260928 14:53:49.319227 28655 raft_consensus.cc:2454] T 9a9ae442e48345aa85ba9b00957c96e0 P bf2ace9ae64b49d19d11dd8b759ded4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfdbd4a7b6004d4db726b89031018cb8 in term 1.
I20260928 14:53:49.319356 28729 leader_election.cc:302] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19a78e3744bb4e3cbb12de42595495ac, bfdbd4a7b6004d4db726b89031018cb8; no voters:
I20260928 14:53:49.319468 29061 raft_consensus.cc:2781] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:49.319609 29061 raft_consensus.cc:686] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8 [term 1 LEADER]: Becoming Leader. State: Replica: bfdbd4a7b6004d4db726b89031018cb8, State: Running, Role: LEADER
I20260928 14:53:49.319670 29061 consensus_queue.cc:237] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } }
I20260928 14:53:49.320308 28498 catalog_manager.cc:5495] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bfdbd4a7b6004d4db726b89031018cb8 (127.20.116.130). New cstate: current_term: 1 leader_uuid: "bfdbd4a7b6004d4db726b89031018cb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfdbd4a7b6004d4db726b89031018cb8" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 45141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 } health_report { overall_health: UNKNOWN } } }
I20260928 14:53:49.329419 29005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:49.327395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57778 (local address 127.20.116.130:45141)
0928 14:53:49.327439 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:49.327441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:49.327468 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:53:49.327532 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:49.327533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:49.327542 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:49.327581 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:53:49.327704 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:49.328163 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:49.328894 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:49.329068 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:49.329119 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:49.329234 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:53:49.329259 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:53:49.329292 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:49.329345 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 14:53:49.329360 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:53:49.331768 28935 raft_consensus.cc:1261] T 9a9ae442e48345aa85ba9b00957c96e0 P 19a78e3744bb4e3cbb12de42595495ac [term 1 FOLLOWER]: Refusing update from remote peer bfdbd4a7b6004d4db726b89031018cb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:49.331773 28655 raft_consensus.cc:1261] T 9a9ae442e48345aa85ba9b00957c96e0 P bf2ace9ae64b49d19d11dd8b759ded4e [term 1 FOLLOWER]: Refusing update from remote peer bfdbd4a7b6004d4db726b89031018cb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:53:49.331987 29038 consensus_queue.cc:1035] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf2ace9ae64b49d19d11dd8b759ded4e" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:49.332129 29061 consensus_queue.cc:1035] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a78e3744bb4e3cbb12de42595495ac" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:53:49.341859 28497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:49.341977 28497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:53:49.342660 28497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ac485d6034446998f660928ada1907b: Sending DeleteTablet for 3 replicas of tablet 9a9ae442e48345aa85ba9b00957c96e0
I20260928 14:53:49.342887 28635 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a9ae442e48345aa85ba9b00957c96e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:49 UTC) from {username='oryx', principal='oryx/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:43376
I20260928 14:53:49.342952 28915 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a9ae442e48345aa85ba9b00957c96e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:49 UTC) from {username='oryx', principal='oryx/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:44284
I20260928 14:53:49.342988 29031 tablet_replica.cc:330] T 9a9ae442e48345aa85ba9b00957c96e0 P bf2ace9ae64b49d19d11dd8b759ded4e: stopping tablet replica
I20260928 14:53:49.343032 29031 raft_consensus.cc:2229] T 9a9ae442e48345aa85ba9b00957c96e0 P bf2ace9ae64b49d19d11dd8b759ded4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:49.343042 29032 tablet_replica.cc:330] T 9a9ae442e48345aa85ba9b00957c96e0 P 19a78e3744bb4e3cbb12de42595495ac: stopping tablet replica
I20260928 14:53:49.343070 29031 raft_consensus.cc:2258] T 9a9ae442e48345aa85ba9b00957c96e0 P bf2ace9ae64b49d19d11dd8b759ded4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:49.343029 28775 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a9ae442e48345aa85ba9b00957c96e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:53:49 UTC) from {username='oryx', principal='oryx/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:57758
I20260928 14:53:49.343109 29032 raft_consensus.cc:2229] T 9a9ae442e48345aa85ba9b00957c96e0 P 19a78e3744bb4e3cbb12de42595495ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:53:49.343148 29032 raft_consensus.cc:2258] T 9a9ae442e48345aa85ba9b00957c96e0 P 19a78e3744bb4e3cbb12de42595495ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:49.343149 29033 tablet_replica.cc:330] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8: stopping tablet replica
I20260928 14:53:49.343204 29033 raft_consensus.cc:2229] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:53:49.343267 29033 raft_consensus.cc:2258] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:53:49.343257 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 28570
I20260928 14:53:49.344121 29033 ts_tablet_manager.cc:1905] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:49.344120 29032 ts_tablet_manager.cc:1905] T 9a9ae442e48345aa85ba9b00957c96e0 P 19a78e3744bb4e3cbb12de42595495ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:53:49.345777 29033 ts_tablet_manager.cc:1918] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:53:49.345837 29033 log.cc:1196] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/9a9ae442e48345aa85ba9b00957c96e0
I20260928 14:53:49.345829 29032 ts_tablet_manager.cc:1918] T 9a9ae442e48345aa85ba9b00957c96e0 P 19a78e3744bb4e3cbb12de42595495ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:53:49.345889 29032 log.cc:1196] T 9a9ae442e48345aa85ba9b00957c96e0 P 19a78e3744bb4e3cbb12de42595495ac: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/9a9ae442e48345aa85ba9b00957c96e0
I20260928 14:53:49.346052 29033 ts_tablet_manager.cc:1939] T 9a9ae442e48345aa85ba9b00957c96e0 P bfdbd4a7b6004d4db726b89031018cb8: Deleting consensus metadata
I20260928 14:53:49.346112 29032 ts_tablet_manager.cc:1939] T 9a9ae442e48345aa85ba9b00957c96e0 P 19a78e3744bb4e3cbb12de42595495ac: Deleting consensus metadata
I20260928 14:53:49.346575 28485 catalog_manager.cc:4841] TS bfdbd4a7b6004d4db726b89031018cb8 (127.20.116.130:45141): tablet 9a9ae442e48345aa85ba9b00957c96e0 (table test-table [id=73d57a7ed542482db0708ebc85e7c739]) successfully deleted
I20260928 14:53:49.346684 28485 catalog_manager.cc:4841] TS 19a78e3744bb4e3cbb12de42595495ac (127.20.116.131:36181): tablet 9a9ae442e48345aa85ba9b00957c96e0 (table test-table [id=73d57a7ed542482db0708ebc85e7c739]) successfully deleted
W20260928 14:53:49.349705 28482 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.116.129:34741 (error 108)
I20260928 14:53:49.350271 29092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:49.350081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:34741 (local address 127.0.0.1:43380)
0928 14:53:49.350173 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:49.350218 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.129:34741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":45,"threads_started":1}
I20260928 14:53:49.350374 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 28710
W20260928 14:53:49.350419 28482 catalog_manager.cc:4584] TS bf2ace9ae64b49d19d11dd8b759ded4e (127.20.116.129:34741): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a9ae442e48345aa85ba9b00957c96e0: Network error: Client connection negotiation failed: client connection to 127.20.116.129:34741: connect: Connection refused (error 111)
I20260928 14:53:49.357067 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 28850
I20260928 14:53:49.363894 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 28466
2026-09-28T14:53:49Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2813 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:53:49 dist-test-slave-pxl6 krb5kdc[29095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:53:49 dist-test-slave-pxl6 krb5kdc[29095](info): set up 2 sockets
Sep 28 14:53:49 dist-test-slave-pxl6 krb5kdc[29095](info): commencing operation
krb5kdc: starting...
W20260928 14:53:50.420315 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:53:50 dist-test-slave-pxl6 krb5kdc[29095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:53:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:53:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:53:50.546594 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:35003
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:35003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:50.631466 29108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:50.631716 29108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:50.631758 29108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:50.632993 29108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:53:50.633106 29108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:50.633133 29108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:50.633160 29108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:53:50.633229 29108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:53:50.634961 29108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:35003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:35003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:50.635272 29108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:50.635516 29108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:50.640576 29113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:50.640741 29114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:50.640820 29108 server_base.cc:1034] running on GCE node
W20260928 14:53:50.640576 29116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:50.641155 29108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:50.641498 29108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:50.642648 29108 hybrid_clock.cc:648] HybridClock initialized: now 1790607230642595 us; error 60 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T14:53:50Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1279 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:53:50 dist-test-slave-pxl6 krb5kdc[29121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:53:50 dist-test-slave-pxl6 krb5kdc[29121](info): set up 2 sockets
Sep 28 14:53:50 dist-test-slave-pxl6 krb5kdc[29121](info): commencing operation
krb5kdc: starting...
W20260928 14:53:51.684356 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:53:51 dist-test-slave-pxl6 krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:53:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:53:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:53:51.813467 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:34953
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:41871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:34953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:51.895949 29134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:51.896173 29134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:51.896196 29134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:51.897396 29134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:53:51.897444 29134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:51.897459 29134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:51.897471 29134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:53:51.897512 29134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:53:51.899051 29134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:41871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:34953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:34953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:51.899319 29134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:51.899552 29134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:51.904992 29140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:51.905023 29139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:51.904992 29142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:51.905218 29134 server_base.cc:1034] running on GCE node
I20260928 14:53:51.905457 29134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:51.905812 29134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:51.907024 29134 hybrid_clock.cc:648] HybridClock initialized: now 1790607231906990 us; error 31 us; skew 500 ppm
Sep 28 14:53:51 dist-test-slave-pxl6 krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607231, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:51.914741 29134 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:53:51.915284 29134 webserver.cc:460] Webserver started at http://127.20.116.190:34685/ using document root <none> and password file <none>
I20260928 14:53:51.915540 29134 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:51.915611 29134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:51.915766 29134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:51.916915 29134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "4fddca2368d143d9ae04e8e4754c6b6b"
format_stamp: "Formatted at 2026-09-28 14:53:51 on dist-test-slave-pxl6"
server_key: "3838472657848bf220fec3db1b066aa3"
server_key_iv: "8216b915cac82bcf7de13e988bcbcd10"
server_key_version: "encryptionkey@0"
I20260928 14:53:51.917286 29134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "4fddca2368d143d9ae04e8e4754c6b6b"
format_stamp: "Formatted at 2026-09-28 14:53:51 on dist-test-slave-pxl6"
server_key: "3838472657848bf220fec3db1b066aa3"
server_key_iv: "8216b915cac82bcf7de13e988bcbcd10"
server_key_version: "encryptionkey@0"
I20260928 14:53:51.918788 29134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:53:51.919605 29149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:51.919800 29134 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:51.919910 29134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "4fddca2368d143d9ae04e8e4754c6b6b"
format_stamp: "Formatted at 2026-09-28 14:53:51 on dist-test-slave-pxl6"
server_key: "3838472657848bf220fec3db1b066aa3"
server_key_iv: "8216b915cac82bcf7de13e988bcbcd10"
server_key_version: "encryptionkey@0"
I20260928 14:53:51.919986 29134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:51.934960 29134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:51.937498 29134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:51.937693 29134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:51.942086 29134 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:34953
I20260928 14:53:51.942126 29201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:34953 every 8 connection(s)
I20260928 14:53:51.942515 29134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:53:51.943449 29202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:53:51.945750 29202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b: Bootstrap starting.
I20260928 14:53:51.946312 29202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b: Neither blocks nor log segments found. Creating new log.
I20260928 14:53:51.946630 29202 log.cc:824] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:51.947304 29202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b: No bootstrap required, opened a new log
I20260928 14:53:51.948652 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 29134
I20260928 14:53:51.948776 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:53:51.948975 20946 external_mini_cluster.cc:1428] Setting key 12126d0c7daea1d80ad4e9f1312c4089
I20260928 14:53:51.949348 29202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fddca2368d143d9ae04e8e4754c6b6b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34953 } }
I20260928 14:53:51.949558 29202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:53:51.949648 29202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fddca2368d143d9ae04e8e4754c6b6b, State: Initialized, Role: FOLLOWER
I20260928 14:53:51.949802 29202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fddca2368d143d9ae04e8e4754c6b6b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34953 } }
I20260928 14:53:51.949885 29202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:53:51.949909 29202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:53:51.949963 29202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:53:51.950738 29202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fddca2368d143d9ae04e8e4754c6b6b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34953 } }
I20260928 14:53:51.950871 29202 leader_election.cc:302] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fddca2368d143d9ae04e8e4754c6b6b; no voters:
I20260928 14:53:51.951047 29202 leader_election.cc:288] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:53:51.951085 29207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:53:51.951277 29207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [term 1 LEADER]: Becoming Leader. State: Replica: 4fddca2368d143d9ae04e8e4754c6b6b, State: Running, Role: LEADER
I20260928 14:53:51.951328 29202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:51.951403 29207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fddca2368d143d9ae04e8e4754c6b6b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34953 } }
I20260928 14:53:51.951781 29207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fddca2368d143d9ae04e8e4754c6b6b. Latest consensus state: current_term: 1 leader_uuid: "4fddca2368d143d9ae04e8e4754c6b6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fddca2368d143d9ae04e8e4754c6b6b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34953 } } }
I20260928 14:53:51.951875 29207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:51.951790 29208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fddca2368d143d9ae04e8e4754c6b6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fddca2368d143d9ae04e8e4754c6b6b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34953 } } }
I20260928 14:53:51.952025 29208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:51.952790 29215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:53:51.953334 29215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:53:51.955122 29215 catalog_manager.cc:1347] Generated new cluster ID: fdaa1b39c5e540c2b0aafe0c3df13560
I20260928 14:53:51.955174 29215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:53:51 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:51.960544 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:51.949611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52364 (local address 127.20.116.190:34953)
0928 14:53:51.949829 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:51.949837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:51.949867 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:53:51.950047 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:51.950060 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:51.950108 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:51.950218 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:51.951077 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:51.952592 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:51.953567 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:51.954041 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:51.954156 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:51.956832 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:51.956835 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:51.956836 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:51.956849 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:51.958658 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:51.959027 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:51.959030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:51.959031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:51.959150 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:51.959376 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:51.959377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:51.959377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:51.959638 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:51.960220 (+ 582us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:51.960281 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 14:53:51.960391 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":106,"threads_started":1}
I20260928 14:53:51.968329 29215 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:53:51.968938 29215 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:53:51.978302 29215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b: Generated new TSK 0
I20260928 14:53:51.978560 29215 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
I20260928 14:53:52.014346 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34953
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:41871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:52.123582 29228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:52.123875 29228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:52.123920 29228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:52.125238 29228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:52.125316 29228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:52.125365 29228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:53:52.127038 29228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:41871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:52.127336 29228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:52.127591 29228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:52.132957 29234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:52.133013 29236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:52.132966 29233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:52.133196 29228 server_base.cc:1034] running on GCE node
I20260928 14:53:52.133391 29228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:52.133700 29228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:52.134876 29228 hybrid_clock.cc:648] HybridClock initialized: now 1790607232134859 us; error 30 us; skew 500 ppm
Sep 28 14:53:52 dist-test-slave-pxl6 krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607232, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:52.137558 29228 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:53:52.138075 29228 webserver.cc:460] Webserver started at http://127.20.116.129:41697/ using document root <none> and password file <none>
I20260928 14:53:52.138290 29228 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:52.138332 29228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:52.138422 29228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:52.139531 29228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "ec12fbbb721a44d3a355043d18535663"
format_stamp: "Formatted at 2026-09-28 14:53:52 on dist-test-slave-pxl6"
server_key: "11fc4223206a155cb905ab9c306302d1"
server_key_iv: "8be916bc983fa7148ce9654362836757"
server_key_version: "encryptionkey@0"
I20260928 14:53:52.139886 29228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "ec12fbbb721a44d3a355043d18535663"
format_stamp: "Formatted at 2026-09-28 14:53:52 on dist-test-slave-pxl6"
server_key: "11fc4223206a155cb905ab9c306302d1"
server_key_iv: "8be916bc983fa7148ce9654362836757"
server_key_version: "encryptionkey@0"
I20260928 14:53:52.141283 29228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:53:52.142033 29243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:52.142247 29228 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:52.142340 29228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "ec12fbbb721a44d3a355043d18535663"
format_stamp: "Formatted at 2026-09-28 14:53:52 on dist-test-slave-pxl6"
server_key: "11fc4223206a155cb905ab9c306302d1"
server_key_iv: "8be916bc983fa7148ce9654362836757"
server_key_version: "encryptionkey@0"
I20260928 14:53:52.142398 29228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:52.172354 29228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:52.175140 29228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:52.175272 29228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:52.175477 29228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:52.175846 29228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:52.175884 29228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:52.175909 29228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:52.175959 29228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:52.181876 29228 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:40533
I20260928 14:53:52.181970 29356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:40533 every 8 connection(s)
I20260928 14:53:52.182314 29228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:53:52 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607232, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:52.190783 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 29228
I20260928 14:53:52.190912 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:53:52.191133 20946 external_mini_cluster.cc:1428] Setting key 3bd668090a403f76932f81b61a4928fb
I20260928 14:53:52.193004 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:52.183426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:56313 (local address 127.20.116.190:34953)
0928 14:53:52.183510 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:52.183512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:52.183730 (+ 218us) server_negotiation.cc:407] Connection header received
0928 14:53:52.184237 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:52.184238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:52.184252 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:52.184319 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:52.185338 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.185849 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.186978 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.187194 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.187255 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:52.189582 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:52.189583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:52.189584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:52.189591 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:52.191342 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:52.191878 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:52.191879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:52.191879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:52.191945 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:52.192264 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:52.192266 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:52.192266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:52.192401 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:52.192487 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:52.192800 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 14:53:52.192907 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:53:52.193234 29359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:52.183377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.129:56313)
0928 14:53:52.183581 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:52.183635 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:52.184146 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:52.184401 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:52.184405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:52.185003 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:52.185232 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:52.185237 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.185970 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.185971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:52.186891 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:52.186894 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.187285 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.187286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:52.187425 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:52.187954 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:52.187964 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:52.189483 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:52.191429 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:52.191439 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:52.191441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:52.191776 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:52.192035 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:52.192036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:52.192036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:52.192175 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:52.192478 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:52.192486 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:52.192650 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 14:53:52.192765 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 14:53:52.192947 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":123,"threads_started":1}
I20260928 14:53:52.193890 29357 heartbeater.cc:347] Connected to a master server at 127.20.116.190:34953
I20260928 14:53:52.194029 29357 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:52.194228 29357 heartbeater.cc:511] Master 127.20.116.190:34953 requested a full tablet report, sending...
I20260928 14:53:52.194831 29166 ts_manager.cc:194] Registered new tserver with Master: ec12fbbb721a44d3a355043d18535663 (127.20.116.129:40533)
I20260928 14:53:52.195580 29166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:56313
WARNING: no policy specified for kudu/127.20.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
I20260928 14:53:52.221899 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34953
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:41871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:52.304677 29362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:52.304915 29362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:52.304951 29362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:52.306125 29362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:52.306193 29362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:52.306239 29362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:53:52.307720 29362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:41871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:52.307992 29362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:52.308231 29362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:52.313519 29370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:52.313506 29367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:52.313601 29362 server_base.cc:1034] running on GCE node
W20260928 14:53:52.313506 29368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:52.314007 29362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:52.314268 29362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:52.315469 29362 hybrid_clock.cc:648] HybridClock initialized: now 1790607232315448 us; error 39 us; skew 500 ppm
Sep 28 14:53:52 dist-test-slave-pxl6 krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607232, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:52.318275 29362 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:53:52.318827 29362 webserver.cc:460] Webserver started at http://127.20.116.130:40469/ using document root <none> and password file <none>
I20260928 14:53:52.319063 29362 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:52.319110 29362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:52.319281 29362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:52.320356 29362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "1695b53f48734164ba6759e8f1e05a96"
format_stamp: "Formatted at 2026-09-28 14:53:52 on dist-test-slave-pxl6"
server_key: "37863c285cef83a8f0d7444635876561"
server_key_iv: "2df85c661bc9e70921226f6b15676891"
server_key_version: "encryptionkey@0"
I20260928 14:53:52.320725 29362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "1695b53f48734164ba6759e8f1e05a96"
format_stamp: "Formatted at 2026-09-28 14:53:52 on dist-test-slave-pxl6"
server_key: "37863c285cef83a8f0d7444635876561"
server_key_iv: "2df85c661bc9e70921226f6b15676891"
server_key_version: "encryptionkey@0"
I20260928 14:53:52.322106 29362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:53:52.323119 29377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:52.323318 29362 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:53:52.323413 29362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "1695b53f48734164ba6759e8f1e05a96"
format_stamp: "Formatted at 2026-09-28 14:53:52 on dist-test-slave-pxl6"
server_key: "37863c285cef83a8f0d7444635876561"
server_key_iv: "2df85c661bc9e70921226f6b15676891"
server_key_version: "encryptionkey@0"
I20260928 14:53:52.323508 29362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:52.349547 29362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:52.352309 29362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:52.352509 29362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:52.352841 29362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:52.353196 29362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:52.353255 29362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:52.353289 29362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:52.353331 29362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:52.359164 29362 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:43243
I20260928 14:53:52.359223 29490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:43243 every 8 connection(s)
I20260928 14:53:52.359589 29362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:53:52 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607232, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:52.367359 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 29362
I20260928 14:53:52.367501 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:53:52.367815 20946 external_mini_cluster.cc:1428] Setting key 1dac160276c5a982dafd6e6c1fad4f4b
I20260928 14:53:52.369362 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:52.360330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:52005 (local address 127.20.116.190:34953)
0928 14:53:52.360378 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:52.360381 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:52.360596 (+ 215us) server_negotiation.cc:407] Connection header received
0928 14:53:52.361098 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:52.361099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:52.361109 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:52.361173 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:52.362151 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.362730 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.363748 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.363946 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.363998 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:52.366081 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:52.366083 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:52.366083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:52.366090 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:52.367628 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:52.368182 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:52.368183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:52.368184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:52.368248 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:52.368548 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:52.368549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:52.368550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:52.368760 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:52.368822 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:52.369215 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 14:53:52.369296 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:53:52.369688 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:52.360291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.130:52005)
0928 14:53:52.360455 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:52.360498 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:52.361001 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:52.361255 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:52.361259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:52.361838 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:52.362068 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:52.362073 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.362807 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.362807 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:52.363644 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:52.363647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.364025 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.364026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:52.364158 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:52.364621 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:52.364629 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:52.365992 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:52.367716 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:52.367725 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:52.367727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:52.368103 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:52.368315 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:52.368316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:52.368316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:52.368451 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:52.368852 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:52.368860 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:52.369047 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 14:53:52.369193 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 14:53:52.369388 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260928 14:53:52.370246 29491 heartbeater.cc:347] Connected to a master server at 127.20.116.190:34953
I20260928 14:53:52.370360 29491 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:52.370605 29491 heartbeater.cc:511] Master 127.20.116.190:34953 requested a full tablet report, sending...
I20260928 14:53:52.371059 29166 ts_manager.cc:194] Registered new tserver with Master: 1695b53f48734164ba6759e8f1e05a96 (127.20.116.130:43243)
I20260928 14:53:52.371600 29166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:52005
WARNING: no policy specified for kudu/127.20.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
I20260928 14:53:52.398890 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34953
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:41871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:52.483935 29496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:52.484148 29496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:52.484174 29496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:52.485392 29496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:52.485438 29496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:52.485466 29496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:53:52.487028 29496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:41871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:52.487287 29496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:52.487577 29496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:52.492749 29501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:52.492807 29504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:52.492767 29502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:52.493023 29496 server_base.cc:1034] running on GCE node
I20260928 14:53:52.493291 29496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:52.493573 29496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:52.494745 29496 hybrid_clock.cc:648] HybridClock initialized: now 1790607232494747 us; error 43 us; skew 500 ppm
Sep 28 14:53:52 dist-test-slave-pxl6 krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607232, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:52.497351 29496 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:53:52.497843 29496 webserver.cc:460] Webserver started at http://127.20.116.131:39875/ using document root <none> and password file <none>
I20260928 14:53:52.498056 29496 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:52.498097 29496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:52.498189 29496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:53:52.499326 29496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "993322af7c994349a073fa9a1493fa65"
format_stamp: "Formatted at 2026-09-28 14:53:52 on dist-test-slave-pxl6"
server_key: "3bbc365260e4832645b4242804695522"
server_key_iv: "42d0fe119184fc490e4bd609aea3ff9e"
server_key_version: "encryptionkey@0"
I20260928 14:53:52.499642 29496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "993322af7c994349a073fa9a1493fa65"
format_stamp: "Formatted at 2026-09-28 14:53:52 on dist-test-slave-pxl6"
server_key: "3bbc365260e4832645b4242804695522"
server_key_iv: "42d0fe119184fc490e4bd609aea3ff9e"
server_key_version: "encryptionkey@0"
I20260928 14:53:52.500974 29496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:53:52.501915 29511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:52.502112 29496 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:52.502205 29496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "993322af7c994349a073fa9a1493fa65"
format_stamp: "Formatted at 2026-09-28 14:53:52 on dist-test-slave-pxl6"
server_key: "3bbc365260e4832645b4242804695522"
server_key_iv: "42d0fe119184fc490e4bd609aea3ff9e"
server_key_version: "encryptionkey@0"
I20260928 14:53:52.502266 29496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:52.514694 29496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:52.516919 29496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:52.517037 29496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:52.517248 29496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:52.517604 29496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:52.517642 29496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:52.517666 29496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:52.517715 29496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:52.523888 29496 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:39463
I20260928 14:53:52.523926 29624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:39463 every 8 connection(s)
I20260928 14:53:52.524325 29496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:53:52 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607232, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:53:52.533738 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:52.525070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:36471 (local address 127.20.116.190:34953)
0928 14:53:52.525134 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:52.525136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:52.525355 (+ 219us) server_negotiation.cc:407] Connection header received
0928 14:53:52.525921 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:52.525922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:52.525936 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:52.525995 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:52.527087 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.527671 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.528687 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.528896 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.528984 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:52.531250 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:52.531251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:52.531252 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:52.531258 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:52.532457 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:52.532867 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:52.532867 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:52.532868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:52.532920 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:52.533135 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:52.533135 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:52.533136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:52.533237 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:52.533288 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:52.533516 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 14:53:52.533673 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:53:52.533865 29627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:52.525043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.131:36471)
0928 14:53:52.525226 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:52.525269 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:52.525807 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:52.526075 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:52.526080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:52.526771 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:52.526973 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:52.526979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.527765 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.527766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:52.528592 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:52.528594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.528995 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.528996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:52.529133 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:52.529714 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:52.529726 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:52.531161 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:53:52.532532 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:52.532538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:52.532539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:52.532795 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:52.532980 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:53:52.532980 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:53:52.532981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:53:52.533075 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:53:52.533290 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:53:52.533294 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:53:52.533407 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 14:53:52.533488 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 14:53:52.533650 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260928 14:53:52.534180 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 29496
I20260928 14:53:52.534303 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:53:52.534427 29625 heartbeater.cc:347] Connected to a master server at 127.20.116.190:34953
I20260928 14:53:52.534551 29625 heartbeater.cc:464] Registering TS with master...
I20260928 14:53:52.534523 20946 external_mini_cluster.cc:1428] Setting key 11961c784acea90c6f9e0e022e437f08
I20260928 14:53:52.534752 29625 heartbeater.cc:511] Master 127.20.116.190:34953 requested a full tablet report, sending...
I20260928 14:53:52.535152 29166 ts_manager.cc:194] Registered new tserver with Master: 993322af7c994349a073fa9a1493fa65 (127.20.116.131:39463)
I20260928 14:53:52.535382 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:53:52.535624 29166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:36471
WARNING: no policy specified for oryx/127.20.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.129.keytab.
Entry for principal oryx/127.20.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.129.keytab.
WARNING: no policy specified for oryx/127.20.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.130.keytab.
Entry for principal oryx/127.20.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.130.keytab.
WARNING: no policy specified for oryx/127.20.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.131.keytab.
Entry for principal oryx/127.20.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.131.keytab.
I20260928 14:53:52.611372 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 29228
I20260928 14:53:52.616425 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 29362
I20260928 14:53:52.620899 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 29496
I20260928 14:53:52.625515 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 29134
I20260928 14:53:52.631302 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:34953
--webserver_interface=127.20.116.190
--webserver_port=34685
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:41871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:34953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:53:52.716009 29634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:52.716230 29634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:52.716254 29634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:52.717482 29634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:53:52.717530 29634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:52.717545 29634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:52.717558 29634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:53:52.717571 29634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:53:52.719159 29634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:41871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:34953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:34953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=34685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:52.719447 29634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:52.719774 29634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:52.724820 29639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:52.724836 29642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:52.724871 29634 server_base.cc:1034] running on GCE node
W20260928 14:53:52.724838 29640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:52.725320 29634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:52.725687 29634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:52.726854 29634 hybrid_clock.cc:648] HybridClock initialized: now 1790607232726829 us; error 37 us; skew 500 ppm
Sep 28 14:53:52 dist-test-slave-pxl6 krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607232, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:52.729427 29634 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:53:52.729946 29634 webserver.cc:460] Webserver started at http://127.20.116.190:34685/ using document root <none> and password file <none>
I20260928 14:53:52.730180 29634 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:52.730262 29634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:52.731734 29634 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:52.732429 29649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:52.732683 29634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:52.732789 29634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "4fddca2368d143d9ae04e8e4754c6b6b"
format_stamp: "Formatted at 2026-09-28 14:53:51 on dist-test-slave-pxl6"
server_key: "3838472657848bf220fec3db1b066aa3"
server_key_iv: "8216b915cac82bcf7de13e988bcbcd10"
server_key_version: "encryptionkey@0"
I20260928 14:53:52.733165 29634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:52.753044 29634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:52.755709 29634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:52.755887 29634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:52.760053 29634 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:34953
I20260928 14:53:52.760090 29701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:34953 every 8 connection(s)
I20260928 14:53:52.760479 29634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:53:52.761942 29702 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:53:52.762603 29702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b: Bootstrap starting.
I20260928 14:53:52.766767 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 29634
I20260928 14:53:52.767064 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:40533
--local_ip_for_outbound_sockets=127.20.116.129
--tserver_master_addrs=127.20.116.190:34953
--webserver_port=41697
--webserver_interface=127.20.116.129
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:41871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.116.129
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:53:52.771552 29702 log.cc:824] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b: Log is configured to *not* fsync() on all Append() calls
I20260928 14:53:52.773314 29702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:53:52.773629 29702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b: Bootstrap complete.
I20260928 14:53:52.776046 29702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fddca2368d143d9ae04e8e4754c6b6b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34953 } }
I20260928 14:53:52.776283 29702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fddca2368d143d9ae04e8e4754c6b6b, State: Initialized, Role: FOLLOWER
I20260928 14:53:52.776455 29702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fddca2368d143d9ae04e8e4754c6b6b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34953 } }
I20260928 14:53:52.776548 29702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:53:52.776620 29702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:53:52.776682 29702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:53:52.777740 29702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fddca2368d143d9ae04e8e4754c6b6b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34953 } }
I20260928 14:53:52.777833 29702 leader_election.cc:302] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fddca2368d143d9ae04e8e4754c6b6b; no voters:
I20260928 14:53:52.778053 29702 leader_election.cc:288] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:53:52.778095 29706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:53:52.778263 29706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [term 2 LEADER]: Becoming Leader. State: Replica: 4fddca2368d143d9ae04e8e4754c6b6b, State: Running, Role: LEADER
I20260928 14:53:52.778340 29702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:53:52.778422 29706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fddca2368d143d9ae04e8e4754c6b6b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34953 } }
I20260928 14:53:52.778728 29706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fddca2368d143d9ae04e8e4754c6b6b. Latest consensus state: current_term: 2 leader_uuid: "4fddca2368d143d9ae04e8e4754c6b6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fddca2368d143d9ae04e8e4754c6b6b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34953 } } }
I20260928 14:53:52.778820 29706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:52.778980 29707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4fddca2368d143d9ae04e8e4754c6b6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fddca2368d143d9ae04e8e4754c6b6b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34953 } } }
I20260928 14:53:52.779049 29707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b [sys.catalog]: This master's current role is: LEADER
I20260928 14:53:52.779249 29713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:53:52.779765 29713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:53:52.779896 29713 catalog_manager.cc:1259] Loaded cluster ID: fdaa1b39c5e540c2b0aafe0c3df13560
I20260928 14:53:52.780119 29713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:53:52.781639 29713 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:53:52.781837 29713 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4fddca2368d143d9ae04e8e4754c6b6b: Loaded TSK: 0
I20260928 14:53:52.782047 29713 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:53:52.853012 29704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:52.853227 29704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:52.853251 29704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:52.854449 29704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:52.854491 29704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:52.854518 29704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:53:52.856045 29704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:41871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.129.keytab
--principal=oryx/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:40533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=41697
--tserver_master_addrs=127.20.116.190:34953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:52.856290 29704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:52.856561 29704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:52.862650 29728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:52.862721 29729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:52.862830 29704 server_base.cc:1034] running on GCE node
W20260928 14:53:52.862658 29731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:52.863180 29704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:52.863474 29704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:52.864669 29704 hybrid_clock.cc:648] HybridClock initialized: now 1790607232864657 us; error 29 us; skew 500 ppm
Sep 28 14:53:52 dist-test-slave-pxl6 krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607232, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:52.867484 29704 init.cc:377] Logged in from keytab as oryx/127.20.116.129@KRBTEST.COM (short username oryx)
I20260928 14:53:52.868068 29704 webserver.cc:460] Webserver started at http://127.20.116.129:41697/ using document root <none> and password file <none>
I20260928 14:53:52.868395 29704 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:52.868484 29704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:52.870330 29704 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:53:52.871086 29738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:52.871312 29704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:53:52.871410 29704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "ec12fbbb721a44d3a355043d18535663"
format_stamp: "Formatted at 2026-09-28 14:53:52 on dist-test-slave-pxl6"
server_key: "11fc4223206a155cb905ab9c306302d1"
server_key_iv: "8be916bc983fa7148ce9654362836757"
server_key_version: "encryptionkey@0"
I20260928 14:53:52.871860 29704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:52.912693 29704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:52.915016 29704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:52.915196 29704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:52.915498 29704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:52.915889 29704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:52.915943 29704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:52.915985 29704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:52.916013 29704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:52.922209 29704 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:40533
I20260928 14:53:52.922253 29851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:40533 every 8 connection(s)
I20260928 14:53:52.922686 29704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:53:52.922972 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 29704
I20260928 14:53:52.923329 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:43243
--local_ip_for_outbound_sockets=127.20.116.130
--tserver_master_addrs=127.20.116.190:34953
--webserver_port=40469
--webserver_interface=127.20.116.130
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:41871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.116.130
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:53:52 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:52 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:52.932819 29854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:52.923524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.129:33519)
0928 14:53:52.923722 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:52.923775 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:52.924541 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:52.925054 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:52.925058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:52.925631 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:52.925862 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:52.925867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.927694 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.927695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:52.928487 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:52.928489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.929394 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.929395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:52.929497 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:52.930095 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:52.930104 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:52.932660 (+ 2556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
W20260928 14:53:52.933060 29855 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:52.923562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:33519 (local address 127.20.116.190:34953)
0928 14:53:52.923787 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:52.923796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:52.923887 (+ 91us) server_negotiation.cc:407] Connection header received
0928 14:53:52.924784 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:52.924792 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:52.924855 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:52.925008 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:52.925998 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.927553 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.928580 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.929292 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.929422 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:52.932848 (+ 3426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.129:33519: BlockingRecv error: recv got EOF from 127.20.116.129:33519 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":105,"threads_started":1}
W20260928 14:53:52.933101 29852 heartbeater.cc:644] Failed to heartbeat to 127.20.116.190:34953 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.116.190:34953: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Sep 28 14:53:52 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:52 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:52.938470 29857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:52.933334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.129:39705)
0928 14:53:52.933497 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:52.933508 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:52.933570 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:52.933718 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:52.933719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:52.933848 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:52.933891 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:52.933893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.934887 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.934887 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:52.935426 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:52.935428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.935945 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.935947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:52.936009 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:52.936485 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:52.936490 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:52.938402 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":106,"threads_started":1}
W20260928 14:53:52.938608 29855 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:52.933420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:39705 (local address 127.20.116.190:34953)
0928 14:53:52.933481 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:52.933483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:52.933576 (+ 93us) server_negotiation.cc:407] Connection header received
0928 14:53:52.933626 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:52.933627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:52.933644 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:52.933708 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:52.933965 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.934811 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.935497 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.935817 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.935886 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:52.938512 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.129:39705: BlockingRecv error: recv got EOF from 127.20.116.129:39705 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 14:53:52 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:52 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:52.943825 29857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:52.939019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.129:52163)
0928 14:53:52.939075 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:52.939086 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:52.939129 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:52.939286 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:52.939286 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:52.939429 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:52.939464 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:52.939466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.940185 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.940186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:52.940754 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:52.940756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.941182 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.941183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:52.941248 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:52.941787 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:52.941792 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:52.943745 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 14:53:52.943926 29855 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:52.939085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:52163 (local address 127.20.116.190:34953)
0928 14:53:52.939131 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:52.939133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:52.939149 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:53:52.939186 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:52.939187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:52.939211 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:52.939287 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:52.939544 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.940021 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.940847 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:52.941084 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:52.941181 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:52.943839 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.129:52163: BlockingRecv error: recv got EOF from 127.20.116.129:52163 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 14:53:52.944013 29852 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:53:53.010646 29856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:53.010862 29856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:53.010887 29856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:53.012136 29856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:53.012187 29856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:53.012215 29856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:53:53.013712 29856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:41871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.130.keytab
--principal=oryx/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:43243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=40469
--tserver_master_addrs=127.20.116.190:34953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:53.013994 29856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:53.014246 29856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:53.019362 29863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:53.019369 29862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:53.019393 29865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:53.019572 29856 server_base.cc:1034] running on GCE node
I20260928 14:53:53.019786 29856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:53.020026 29856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:53.021236 29856 hybrid_clock.cc:648] HybridClock initialized: now 1790607233021203 us; error 36 us; skew 500 ppm
Sep 28 14:53:53 dist-test-slave-pxl6 krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607233, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:53.023864 29856 init.cc:377] Logged in from keytab as oryx/127.20.116.130@KRBTEST.COM (short username oryx)
I20260928 14:53:53.024339 29856 webserver.cc:460] Webserver started at http://127.20.116.130:40469/ using document root <none> and password file <none>
I20260928 14:53:53.024539 29856 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:53.024636 29856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:53.025939 29856 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:53.026479 29872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:53.026693 29856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:53:53.026782 29856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "1695b53f48734164ba6759e8f1e05a96"
format_stamp: "Formatted at 2026-09-28 14:53:52 on dist-test-slave-pxl6"
server_key: "37863c285cef83a8f0d7444635876561"
server_key_iv: "2df85c661bc9e70921226f6b15676891"
server_key_version: "encryptionkey@0"
I20260928 14:53:53.027104 29856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:53.058032 29856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:53.060535 29856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:53.060657 29856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:53.060842 29856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:53.061182 29856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:53.061219 29856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:53.061242 29856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:53.061288 29856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:53.067631 29856 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:43243
I20260928 14:53:53.067770 29985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:43243 every 8 connection(s)
I20260928 14:53:53.068086 29856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:53:53.069221 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 29856
I20260928 14:53:53.069509 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:39463
--local_ip_for_outbound_sockets=127.20.116.131
--tserver_master_addrs=127.20.116.190:34953
--webserver_port=39875
--webserver_interface=127.20.116.131
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:41871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.116.131
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:53:53 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:53 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:53.076571 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:53.068844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.130:59403)
0928 14:53:53.069026 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:53.069064 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:53.069727 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:53.070004 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:53.070009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:53.070745 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:53.071069 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:53.071077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.071815 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.071816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:53.072718 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:53.072721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.073123 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.073124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:53.073274 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:53.073837 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:53.073847 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:53.076378 (+ 2531us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
W20260928 14:53:53.076579 29855 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:53.068835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:59403 (local address 127.20.116.190:34953)
0928 14:53:53.068907 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:53.068909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:53.069162 (+ 253us) server_negotiation.cc:407] Connection header received
0928 14:53:53.069848 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:53.069850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:53.069868 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:53.069925 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:53.071204 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.071707 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.072823 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.073031 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.073113 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:53.076473 (+ 3360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.130:59403: BlockingRecv error: recv got EOF from 127.20.116.130:59403 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 14:53:53.076860 29986 heartbeater.cc:644] Failed to heartbeat to 127.20.116.190:34953 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.116.190:34953: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Sep 28 14:53:53 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:53 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:53.082242 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:53.077110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.130:53309)
0928 14:53:53.077173 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:53.077185 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:53.077238 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:53.077374 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:53.077374 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:53.077483 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:53.077529 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:53.077530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.078528 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.078585 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:53.079231 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:53.079234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.079743 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.079744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:53.079796 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:53.080351 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:53.080356 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:53.082186 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 14:53:53.082358 29855 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:53.077177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:53309 (local address 127.20.116.190:34953)
0928 14:53:53.077246 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:53.077248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:53.077259 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:53:53.077287 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:53.077288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:53.077300 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:53.077374 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:53.077601 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.078427 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.079306 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.079644 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.079737 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:53.082293 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.130:53309: BlockingRecv error: recv got EOF from 127.20.116.130:53309 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 14:53:53 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:53 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:53.087538 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:53.082514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.130:60613)
0928 14:53:53.082685 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:53.082697 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:53.082777 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:53.082935 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:53.082936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:53.083050 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:53.083081 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:53.083082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.084104 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.084105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:53.084661 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:53.084663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.085131 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.085133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:53.085177 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:53.085720 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:53.085726 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:53.087476 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":117,"threads_started":1}
W20260928 14:53:53.087664 29855 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:53.082601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:60613 (local address 127.20.116.190:34953)
0928 14:53:53.082640 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:53.082642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:53.082743 (+ 101us) server_negotiation.cc:407] Connection header received
0928 14:53:53.082829 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:53.082829 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:53.082840 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:53.082902 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:53.083239 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.083999 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.084748 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.085052 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.085117 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:53.087590 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.130:60613: BlockingRecv error: recv got EOF from 127.20.116.130:60613 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 14:53:53.087705 29986 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:53:53.157430 29989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:53:53.157660 29989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:53:53.157685 29989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:53:53.158927 29989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:53:53.158969 29989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:53:53.158995 29989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:53:53.160430 29989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:41871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/krb5kdc/oryx_127.20.116.131.keytab
--principal=oryx/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:39463
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=39875
--tserver_master_addrs=127.20.116.190:34953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:53:53.160636 29989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:53:53.160828 29989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:53:53.166154 29998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:53:53.166165 29996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:53.166291 29989 server_base.cc:1034] running on GCE node
W20260928 14:53:53.166402 29995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:53:53.166692 29989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:53:53.166973 29989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:53:53.168116 29989 hybrid_clock.cc:648] HybridClock initialized: now 1790607233168098 us; error 31 us; skew 500 ppm
Sep 28 14:53:53 dist-test-slave-pxl6 krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607233, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:53:53.170881 29989 init.cc:377] Logged in from keytab as oryx/127.20.116.131@KRBTEST.COM (short username oryx)
I20260928 14:53:53.171365 29989 webserver.cc:460] Webserver started at http://127.20.116.131:39875/ using document root <none> and password file <none>
I20260928 14:53:53.171651 29989 fs_manager.cc:359] Metadata directory not provided
I20260928 14:53:53.171741 29989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:53:53.173187 29989 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:53.173833 30005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:53.174028 29989 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:53:53.174119 29989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "993322af7c994349a073fa9a1493fa65"
format_stamp: "Formatted at 2026-09-28 14:53:52 on dist-test-slave-pxl6"
server_key: "3bbc365260e4832645b4242804695522"
server_key_iv: "42d0fe119184fc490e4bd609aea3ff9e"
server_key_version: "encryptionkey@0"
I20260928 14:53:53.174443 29989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:53:53.212005 29989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:53:53.214496 29989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:53:53.214655 29989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:53:53.214869 29989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:53:53.215301 29989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:53:53.215340 29989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:53.215363 29989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:53:53.215376 29989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:53:53.221225 29989 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:39463
I20260928 14:53:53.221305 30118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:39463 every 8 connection(s)
I20260928 14:53:53.221678 29989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:53:53.225449 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 29989
Sep 28 14:53:53 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:53 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:53:53.229900 29855 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:53.222466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:49311 (local address 127.20.116.190:34953)
0928 14:53:53.222609 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:53.222612 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:53.222794 (+ 182us) server_negotiation.cc:407] Connection header received
0928 14:53:53.223352 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:53.223354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:53.223367 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:53.223426 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:53.224398 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.224914 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.226149 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.226351 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.226439 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:53.229809 (+ 3370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.131:49311: BlockingRecv error: recv got EOF from 127.20.116.131:49311 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:53:53.229902 30121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:53.222419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.131:49311)
0928 14:53:53.222644 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:53.222692 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:53.223234 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:53.223498 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:53.223502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:53.224095 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:53.224292 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:53.224297 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.225021 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.225023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:53.226036 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:53.226040 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.226453 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.226454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:53.226600 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:53.227136 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:53.227148 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:53.229699 (+ 2551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":146,"threads_started":1}
W20260928 14:53:53.230166 30119 heartbeater.cc:644] Failed to heartbeat to 127.20.116.190:34953 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.116.190:34953: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Sep 28 14:53:53 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:53.234079 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:53.225998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52372 (local address 127.20.116.190:34953)
0928 14:53:53.226132 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:53.226134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:53.226143 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:53:53.226166 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:53.226167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:53.226181 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:53.226261 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:53.226961 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.227719 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.228579 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.228808 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.228863 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:53.230609 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:53:53.230618 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:53:53.230618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:53:53.230635 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:53:53.232447 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:53.232859 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:53.232862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:53.232863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:53.232957 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:53:53.233169 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:53:53.233170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:53:53.233170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:53:53.233476 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:53:53.233715 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 14:53:53.233842 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 14:53:53.233959 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
Sep 28 14:53:53 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:53:53.234915 29855 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:53.230469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:41713 (local address 127.20.116.190:34953)
0928 14:53:53.230563 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:53.230565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:53.230573 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:53:53.230602 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:53.230603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:53.230612 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:53.230662 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:53.230910 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.231375 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.232052 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.232251 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.232350 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:53.234857 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.131:41713: BlockingRecv error: recv got EOF from 127.20.116.131:41713 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:53:53.235121 30121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:53.230409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.131:41713)
0928 14:53:53.230452 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:53.230462 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:53.230520 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:53.230664 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:53.230664 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:53.230794 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:53.230837 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:53.230838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.231469 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.231470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:53.231968 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:53.231970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.232351 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.232352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:53.232416 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:53.232770 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:53.232776 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:53.235059 (+ 2283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
Sep 28 14:53:53 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:53 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:53.239607 30121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:53.235385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.131:53531)
0928 14:53:53.235483 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:53.235492 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:53.235534 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:53.235675 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:53.235676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:53.235786 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:53.235812 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:53.235813 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.236481 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.236482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:53.237010 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:53.237012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.237361 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.237362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:53.237399 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:53.237878 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:53.237883 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:53.239554 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":82}
W20260928 14:53:53.239784 29855 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:53.235446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:53531 (local address 127.20.116.190:34953)
0928 14:53:53.235493 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:53.235494 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:53.235523 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:53:53.235570 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:53.235571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:53.235579 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:53.235660 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:53.235900 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.236417 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.237112 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.237290 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.237353 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:53.239673 (+ 2320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.131:53531: BlockingRecv error: recv got EOF from 127.20.116.131:53531 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 14:53:53.239776 30119 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 14:53:53 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:53 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:53.949541 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:53.944503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.129:39431)
0928 14:53:53.944706 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:53.944719 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:53.944793 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:53.944950 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:53.944951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:53.945142 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:53.945195 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:53.945196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.945885 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.945886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:53.946438 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:53.946440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.946883 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.946884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:53.946941 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:53.947495 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:53.947499 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:53.949472 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":105,"threads_started":1}
W20260928 14:53:53.949704 30125 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:53.944616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:39431 (local address 127.20.116.190:34953)
0928 14:53:53.944728 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:53.944730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:53.944776 (+ 46us) server_negotiation.cc:407] Connection header received
0928 14:53:53.944863 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:53.944863 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:53.944875 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:53.944925 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:53.945269 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.945799 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.946548 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:53.946783 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:53.946842 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:53.949567 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.129:39431: BlockingRecv error: recv got EOF from 127.20.116.129:39431 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":53,"threads_started":1}
Sep 28 14:53:54 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:54 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:54.093060 30126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:54.088169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.130:40295)
0928 14:53:54.088335 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:54.088346 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:54.088407 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:54.088548 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:54.088549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:54.088721 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:54.088773 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:54.088775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:54.089410 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:54.089411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:54.089960 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:54.089962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:54.090349 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:54.090350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:54.090396 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:54.090869 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:54.090874 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:54.092985 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":97,"threads_started":1}
W20260928 14:53:54.093148 30125 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:54.088275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:40295 (local address 127.20.116.190:34953)
0928 14:53:54.088344 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:54.088346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:54.088409 (+ 63us) server_negotiation.cc:407] Connection header received
0928 14:53:54.088452 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:54.088453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:54.088466 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:54.088526 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:54.088841 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:54.089339 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:54.090072 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:54.090269 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:54.090334 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:54.093102 (+ 2768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.130:40295: BlockingRecv error: recv got EOF from 127.20.116.130:40295 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 14:53:54 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:54 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:54.245394 30127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:54.240263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.131:51749)
0928 14:53:54.240438 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:54.240460 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:54.240529 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:54.240689 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:54.240690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:54.240883 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:54.240938 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:54.240940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:54.241612 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:54.241613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:54.242233 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:54.242235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:54.242637 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:54.242639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:54.242698 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:54.243164 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:54.243170 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:54.245321 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":110,"threads_started":1}
W20260928 14:53:54.245570 30125 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:54.240351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:51749 (local address 127.20.116.190:34953)
0928 14:53:54.240424 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:54.240426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:54.240527 (+ 101us) server_negotiation.cc:407] Connection header received
0928 14:53:54.240582 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:54.240583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:54.240597 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:54.240663 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:54.241010 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:54.241529 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:54.242328 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:54.242520 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:54.242603 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:54.245471 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.131:51749: BlockingRecv error: recv got EOF from 127.20.116.131:51749 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 14:53:54 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:54 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:54.955032 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:54.950048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.129:59497)
0928 14:53:54.950240 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:54.950252 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:54.950311 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:54.950450 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:54.950451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:54.950633 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:54.950685 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:54.950687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:54.951365 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:54.951366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:54.951913 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:54.951914 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:54.952306 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:54.952307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:54.952377 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:54.952918 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:54.952923 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:54.954970 (+ 2047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":94,"threads_started":1}
W20260928 14:53:54.955216 30129 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:54.950161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:59497 (local address 127.20.116.190:34953)
0928 14:53:54.950300 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:54.950303 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:54.950312 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:53:54.950361 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:54.950362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:54.950372 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:54.950432 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:54.950766 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:54.951288 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:54.952009 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:54.952216 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:54.952303 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:54.955136 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.129:59497: BlockingRecv error: recv got EOF from 127.20.116.129:59497 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
Sep 28 14:53:55 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:55 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:55.098795 30130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:55.093778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.130:44989)
0928 14:53:55.093951 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:55.093966 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:55.094050 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:55.094231 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:55.094232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:55.094414 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:55.094463 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:55.094465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:55.095205 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:55.095207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:55.095796 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:55.095798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:55.096186 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:55.096187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:55.096248 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:55.096784 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:55.096790 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:55.098699 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
W20260928 14:53:55.098862 30129 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:55.093879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:44989 (local address 127.20.116.190:34953)
0928 14:53:55.093936 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:55.093938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:55.094030 (+ 92us) server_negotiation.cc:407] Connection header received
0928 14:53:55.094126 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:55.094128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:55.094142 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:55.094220 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:55.094568 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:55.095087 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:55.095891 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:55.096104 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:55.096184 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:55.098798 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.130:44989: BlockingRecv error: recv got EOF from 127.20.116.130:44989 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 14:53:55 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:55 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:55.251389 30131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:55.246107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.131:47667)
0928 14:53:55.246303 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:55.246317 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:55.246390 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:55.246619 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:55.246620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:55.246787 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:55.246837 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:55.246839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:55.247532 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:55.247534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:55.248133 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:55.248135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:55.248595 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:55.248596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:55.248649 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:55.249172 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:55.249177 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:55.251290 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
W20260928 14:53:55.251438 30129 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:55.246287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:47667 (local address 127.20.116.190:34953)
0928 14:53:55.246358 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:55.246360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:55.246375 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:53:55.246457 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:55.246458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:55.246471 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:55.246523 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:55.246917 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:55.247424 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:55.248250 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:55.248468 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:55.248546 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:55.251374 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.131:47667: BlockingRecv error: recv got EOF from 127.20.116.131:47667 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 14:53:55 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:55 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:55.960803 30132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:55.955590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.129:39567)
0928 14:53:55.955927 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:55.955940 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:55.956016 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:55.956193 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:55.956194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:55.956366 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:55.956422 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:55.956423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:55.957322 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:55.957324 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:55.957871 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:55.957873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:55.958262 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:55.958263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:55.958330 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:55.958882 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:55.958886 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:55.960701 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"spinlock_wait_cycles":14592,"thread_start_us":142,"threads_started":1}
W20260928 14:53:55.960856 30133 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:55.955714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:39567 (local address 127.20.116.190:34953)
0928 14:53:55.955862 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:55.955865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:55.956061 (+ 196us) server_negotiation.cc:407] Connection header received
0928 14:53:55.956095 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:55.956096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:55.956110 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:55.956166 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:55.956544 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:55.957215 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:55.957956 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:55.958168 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:55.958246 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:55.960785 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.129:39567: BlockingRecv error: recv got EOF from 127.20.116.129:39567 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
Sep 28 14:53:56 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:56 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:56.104650 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:56.099425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.130:53093)
0928 14:53:56.099731 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:56.099748 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:56.099838 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:56.100003 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:56.100004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:56.100198 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:56.100262 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:56.100264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:56.100988 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:56.100990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:56.101643 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:56.101651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:56.102165 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:56.102166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:56.102238 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:56.102769 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:56.102774 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:56.104586 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":197,"threads_started":1}
W20260928 14:53:56.104866 30133 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:56.099652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:53093 (local address 127.20.116.190:34953)
0928 14:53:56.099759 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:56.099762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:56.099814 (+ 52us) server_negotiation.cc:407] Connection header received
0928 14:53:56.099904 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:56.099905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:56.099926 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:56.099995 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:56.100356 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:56.100889 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:56.101773 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:56.102056 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:56.102156 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:56.104789 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.130:53093: BlockingRecv error: recv got EOF from 127.20.116.130:53093 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 14:53:56 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:56 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:56.257033 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:56.251991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.131:52127)
0928 14:53:56.252173 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:56.252186 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:56.252260 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:56.252419 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:56.252419 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:56.252594 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:56.252645 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:56.252647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:56.253314 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:56.253315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:56.254043 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:56.254046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:56.254621 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:56.254622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:56.254674 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:56.255138 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:56.255143 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:56.256940 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":68,"threads_started":1}
W20260928 14:53:56.257099 30133 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:56.252194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:52127 (local address 127.20.116.190:34953)
0928 14:53:56.252275 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:56.252278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:56.252288 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:53:56.252325 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:56.252325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:56.252339 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:56.252389 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:56.252726 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:56.253240 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:56.254227 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:56.254499 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:56.254581 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:56.257037 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.131:52127: BlockingRecv error: recv got EOF from 127.20.116.131:52127 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 14:53:56 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:56 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:56.970136 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:56.961374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.129:43003)
0928 14:53:56.961501 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:56.961512 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:56.961560 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:56.961683 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:56.961683 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:56.961846 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:56.961897 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:56.961899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:56.962599 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:56.962600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:56.963124 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:56.963125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:56.963477 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:56.963478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:56.963517 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:56.968345 (+ 4828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:56.968351 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:56.970073 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
W20260928 14:53:56.970292 30137 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:56.961426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:43003 (local address 127.20.116.190:34953)
0928 14:53:56.961559 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:56.961561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:56.961569 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:53:56.961607 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:56.961608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:56.961619 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:56.961680 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:56.961959 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:56.962484 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:56.963213 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:56.963410 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:56.963464 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:56.970200 (+ 6736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.129:43003: BlockingRecv error: recv got EOF from 127.20.116.129:43003 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
Sep 28 14:53:57 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:57 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:57.110056 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:57.105256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.130:36473)
0928 14:53:57.105384 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:57.105400 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:57.105483 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:57.105659 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:57.105660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:57.105836 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:57.105884 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:57.105885 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:57.106658 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:57.106659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:57.107208 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:57.107210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:57.107594 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:57.107595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:57.107652 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:57.108103 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:57.108108 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:57.109957 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
W20260928 14:53:57.110126 30137 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:57.105313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:36473 (local address 127.20.116.190:34953)
0928 14:53:57.105365 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:57.105367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:57.105463 (+ 96us) server_negotiation.cc:407] Connection header received
0928 14:53:57.105559 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:57.105560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:57.105574 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:57.105611 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:57.105965 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:57.106492 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:57.107311 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:57.107515 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:57.107594 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:57.110060 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.130:36473: BlockingRecv error: recv got EOF from 127.20.116.130:36473 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 14:53:57 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:57 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:57.262428 30139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:57.257649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.131:37299)
0928 14:53:57.257816 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:57.257834 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:57.257895 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:57.258141 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:57.258143 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:57.258377 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:57.258441 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:57.258444 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:57.259212 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:57.259213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:57.259743 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:57.259745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:57.260068 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:57.260069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:57.260109 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:57.260621 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:57.260626 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:57.262366 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":115,"threads_started":1}
W20260928 14:53:57.262575 30137 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:57.257738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:37299 (local address 127.20.116.190:34953)
0928 14:53:57.257814 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:57.257817 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:57.257898 (+ 81us) server_negotiation.cc:407] Connection header received
0928 14:53:57.257995 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:57.257996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:57.258011 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:57.258106 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:57.258555 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:57.259120 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:57.259821 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:57.260002 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:57.260055 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:57.262469 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.131:37299: BlockingRecv error: recv got EOF from 127.20.116.131:37299 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 14:53:57 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:57 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:57.975554 30140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:57.970781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.129:33739)
0928 14:53:57.970951 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:57.970980 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:57.971047 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:57.971174 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:57.971174 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:57.971345 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:57.971398 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:57.971400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:57.972064 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:57.972065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:57.972641 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:57.972643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:57.973026 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:57.973027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:57.973070 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:57.973526 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:57.973530 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:57.975484 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":104,"threads_started":1}
W20260928 14:53:57.975761 30141 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:57.970847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:33739 (local address 127.20.116.190:34953)
0928 14:53:57.970989 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:57.970991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:57.971048 (+ 57us) server_negotiation.cc:407] Connection header received
0928 14:53:57.971092 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:57.971093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:57.971104 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:57.971169 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:57.971461 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:57.971963 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:57.972746 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:57.972948 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:57.973013 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:57.975644 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.129:33739: BlockingRecv error: recv got EOF from 127.20.116.129:33739 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
Sep 28 14:53:58 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:58 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:58.116245 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:58.110723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.130:34349)
0928 14:53:58.110867 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:58.110887 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:58.110966 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:58.111185 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:58.111185 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:58.111380 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:58.111431 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:58.111433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:58.112226 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:58.112227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:58.112819 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:58.112821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:58.113251 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:58.113252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:58.113330 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:58.113901 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:58.113907 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:58.116141 (+ 2234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
W20260928 14:53:58.116592 30141 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:58.110842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:34349 (local address 127.20.116.190:34953)
0928 14:53:58.110912 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:58.110915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:58.110946 (+ 31us) server_negotiation.cc:407] Connection header received
0928 14:53:58.111095 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:58.111096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:58.111110 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:58.111175 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:58.111575 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:58.112125 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:58.112919 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:58.113160 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:58.113250 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:58.116472 (+ 3222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.130:34349: BlockingRecv error: recv got EOF from 127.20.116.130:34349 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 14:53:58 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:58 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:58.267843 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:58.262947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.131:37081)
0928 14:53:58.263106 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:58.263125 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:58.263181 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:58.263313 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:58.263314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:58.263479 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:58.263530 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:58.263532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:58.264296 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:58.264297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:58.264831 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:58.264833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:58.265155 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:58.265156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:58.265201 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:58.265694 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:58.265700 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:58.267778 (+ 2078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
W20260928 14:53:58.267946 30141 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:58.263040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:37081 (local address 127.20.116.190:34953)
0928 14:53:58.263107 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:58.263109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:58.263186 (+ 77us) server_negotiation.cc:407] Connection header received
0928 14:53:58.263226 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:58.263227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:58.263239 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:58.263295 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:58.263591 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:58.264221 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:58.264906 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:58.265087 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:58.265139 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:58.267875 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.131:37081: BlockingRecv error: recv got EOF from 127.20.116.131:37081 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 14:53:58 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:58 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:58.981513 30144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:58.976121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.129:49651)
0928 14:53:58.976379 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:58.976390 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:58.976464 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:58.976669 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:58.976670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:58.976868 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:58.976916 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:58.976919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:58.977641 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:58.977642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:58.978241 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:58.978282 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:58.978741 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:58.978742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:58.978809 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:58.979355 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:58.979361 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:58.981418 (+ 2057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
W20260928 14:53:58.981578 30145 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:58.976236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:49651 (local address 127.20.116.190:34953)
0928 14:53:58.976374 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:58.976376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:58.976459 (+ 83us) server_negotiation.cc:407] Connection header received
0928 14:53:58.976522 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:58.976523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:58.976535 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:58.976696 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:58.977008 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:58.977544 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:58.978351 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:58.978644 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:58.978737 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:58.981508 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.129:49651: BlockingRecv error: recv got EOF from 127.20.116.129:49651 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
Sep 28 14:53:59 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:59 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:59.121783 30146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:59.116957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.130:59209)
0928 14:53:59.117150 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:59.117162 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:59.117231 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:59.117389 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:59.117390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:59.117551 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:59.117606 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:59.117607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:59.118241 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:59.118242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:59.118797 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:59.118799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:59.119141 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:59.119142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:59.119180 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:59.119704 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:59.119709 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:59.121711 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":112,"threads_started":1}
W20260928 14:53:59.121917 30145 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:59.117049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:59209 (local address 127.20.116.190:34953)
0928 14:53:59.117099 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:59.117101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:59.117230 (+ 129us) server_negotiation.cc:407] Connection header received
0928 14:53:59.117309 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:59.117309 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:59.117321 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:59.117379 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:59.117671 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:59.118166 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:59.118873 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:59.119070 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:59.119127 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:59.121809 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.130:59209: BlockingRecv error: recv got EOF from 127.20.116.130:59209 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 14:53:59 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:59 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:59.273241 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:59.268378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.131:49759)
0928 14:53:59.268537 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:59.268551 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:59.268637 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:59.268834 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:59.268836 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:59.269011 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:59.269059 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:59.269061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:59.269768 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:59.269770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:59.270308 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:59.270310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:59.270739 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:59.270741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:59.270808 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:59.271291 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:59.271301 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:59.273143 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
W20260928 14:53:59.273289 30145 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:59.268503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:49759 (local address 127.20.116.190:34953)
0928 14:53:59.268579 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:59.268582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:59.268617 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:53:59.268729 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:59.268730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:59.268751 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:59.268809 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:59.269155 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:59.269678 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:59.270409 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:59.270657 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:59.270728 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:59.273240 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.131:49759: BlockingRecv error: recv got EOF from 127.20.116.131:49759 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 14:53:59 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:53:59 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:53:59.987277 30148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:53:59.982099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.129:33961)
0928 14:53:59.982295 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 14:53:59.982306 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:53:59.982375 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:53:59.982705 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:53:59.982706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:53:59.982895 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:53:59.982948 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:59.982950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:59.983608 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:59.983609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:59.984151 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:53:59.984153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:53:59.984471 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:53:59.984472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:53:59.984511 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:59.985043 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:53:59.985049 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:53:59.987208 (+ 2159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":109,"threads_started":1}
W20260928 14:53:59.987419 30149 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:53:59.982202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:33961 (local address 127.20.116.190:34953)
0928 14:53:59.982330 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 14:53:59.982333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:53:59.982357 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:53:59.982466 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:53:59.982467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:53:59.982482 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:53:59.982577 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:53:59.983027 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:59.983539 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:59.984223 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:53:59.984408 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:53:59.984456 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:53:59.987307 (+ 2851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.129:33961: BlockingRecv error: recv got EOF from 127.20.116.129:33961 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":55,"threads_started":1}
Sep 28 14:54:00 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:00 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:00.127223 30150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:00.122254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.130:40013)
0928 14:54:00.122456 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:00.122468 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:00.122593 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:00.122791 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:00.122792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:00.122977 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:00.123025 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:00.123027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:00.123738 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:00.123739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:00.124275 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:00.124277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:00.124674 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:00.124675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:00.124748 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:00.125230 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:00.125234 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:00.127115 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":121,"threads_started":1}
W20260928 14:54:00.127281 30149 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:00.122338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:40013 (local address 127.20.116.190:34953)
0928 14:54:00.122442 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:00.122445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:00.122545 (+ 100us) server_negotiation.cc:407] Connection header received
0928 14:54:00.122664 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:00.122665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:00.122680 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:00.122783 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:00.123111 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:00.123643 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:00.124360 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:00.124583 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:00.124670 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:00.127210 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.130:40013: BlockingRecv error: recv got EOF from 127.20.116.130:40013 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 14:54:00 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:00 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:00.279179 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:00.273947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.131:39477)
0928 14:54:00.274081 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:00.274120 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:00.274189 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:00.274433 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:00.274434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:00.274775 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:00.274864 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:00.274867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:00.275564 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:00.275566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:00.276107 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:00.276108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:00.276472 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:00.276473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:00.276550 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:00.277046 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:00.277050 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:00.279119 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
W20260928 14:54:00.279404 30149 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:00.274089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:39477 (local address 127.20.116.190:34953)
0928 14:54:00.274162 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:00.274164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:00.274175 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:54:00.274279 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:00.274280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:00.274295 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:00.274354 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:00.274932 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:00.275462 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:00.276191 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:00.276384 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:00.276470 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:00.279344 (+ 2874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.131:39477: BlockingRecv error: recv got EOF from 127.20.116.131:39477 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 14:54:00 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:00 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:00.993145 30152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:00.987768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.129:60867)
0928 14:54:00.987909 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:00.987922 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:00.987988 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:00.988164 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:00.988165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:00.988335 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:00.988386 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:00.988387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:00.989593 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:00.989594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:00.990125 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:00.990127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:00.990626 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:00.990627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:00.990676 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:00.991197 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:00.991202 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:00.993081 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
W20260928 14:54:00.993321 30153 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:00.987901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:60867 (local address 127.20.116.190:34953)
0928 14:54:00.988032 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:00.988034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:00.988050 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:54:00.988076 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:00.988077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:00.988087 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:00.988146 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:00.988451 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:00.989497 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:00.990223 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:00.990447 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:00.990522 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:00.993213 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.129:60867: BlockingRecv error: recv got EOF from 127.20.116.129:60867 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
Sep 28 14:54:01 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:01 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:01.133327 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:01.127793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.130:40105)
0928 14:54:01.127962 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:01.127974 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:01.128045 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:01.128190 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:01.128191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:01.128355 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:01.128404 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:01.128406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:01.129508 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:01.129509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:01.130405 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:01.130408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:01.130875 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:01.130876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:01.130929 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:01.131343 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:01.131347 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:01.133265 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":105,"threads_started":1}
W20260928 14:54:01.133425 30153 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:01.127916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:40105 (local address 127.20.116.190:34953)
0928 14:54:01.127989 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:01.127991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:01.128065 (+ 74us) server_negotiation.cc:407] Connection header received
0928 14:54:01.128096 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:01.128097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:01.128109 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:01.128178 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:01.128471 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:01.129421 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:01.130495 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:01.130769 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:01.130849 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:01.133356 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.130:40105: BlockingRecv error: recv got EOF from 127.20.116.130:40105 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 14:54:01 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:01 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:01.284497 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:01.279716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.131:59001)
0928 14:54:01.279883 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:01.279896 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:01.280000 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:01.280174 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:01.280175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:01.280352 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:01.280399 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:01.280400 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:01.281080 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:01.281081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:01.281624 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:01.281626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:01.282019 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:01.282020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:01.282086 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:01.282559 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:01.282564 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:01.284426 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
W20260928 14:54:01.284639 30153 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:01.279795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:59001 (local address 127.20.116.190:34953)
0928 14:54:01.279886 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:01.279889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:01.279980 (+ 91us) server_negotiation.cc:407] Connection header received
0928 14:54:01.280089 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:01.280090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:01.280110 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:01.280175 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:01.280484 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:01.280987 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:01.281733 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:01.281929 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:01.282016 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:01.284549 (+ 2533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.131:59001: BlockingRecv error: recv got EOF from 127.20.116.131:59001 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 14:54:01 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:01 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:02.005471 30156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:01.993694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.129:38601)
0928 14:54:01.993841 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:01.993853 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:01.993909 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:01.994077 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:01.994077 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:01.994261 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:01.994312 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:01.994314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:01.995018 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:01.995019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:01.995555 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:01.995556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:01.995914 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:01.995915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:01.995961 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:02.002969 (+ 7008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:02.002976 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:02.005407 (+ 2431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
W20260928 14:54:02.005708 30157 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:01.993814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:38601 (local address 127.20.116.190:34953)
0928 14:54:01.993950 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:01.993951 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:01.993959 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:54:01.993982 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:01.993982 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:01.993997 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:01.994055 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:01.994382 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:01.994945 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:01.995644 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:01.995843 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:01.995896 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:02.005603 (+ 9707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.129:38601: BlockingRecv error: recv got EOF from 127.20.116.129:38601 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
Sep 28 14:54:02 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:02 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:02.139144 30158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:02.133860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.130:38951)
0928 14:54:02.134149 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:02.134162 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:02.134240 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:02.134412 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:02.134413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:02.134704 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:02.134752 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:02.134754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:02.135440 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:02.135441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:02.135980 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:02.135982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:02.136434 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:02.136435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:02.136495 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:02.137124 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:02.137130 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:02.139075 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":93,"threads_started":1}
W20260928 14:54:02.139282 30157 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:02.133978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:38951 (local address 127.20.116.190:34953)
0928 14:54:02.134155 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:02.134158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:02.134218 (+ 60us) server_negotiation.cc:407] Connection header received
0928 14:54:02.134324 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:02.134325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:02.134337 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:02.134408 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:02.134843 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:02.135352 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:02.136083 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:02.136356 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:02.136435 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:02.139199 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.130:38951: BlockingRecv error: recv got EOF from 127.20.116.130:38951 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
Sep 28 14:54:02 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:02 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:02.290288 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:02.285002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.131:49987)
0928 14:54:02.285220 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:02.285238 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:02.285329 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:02.285516 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:02.285517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:02.285715 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:02.285766 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:02.285768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:02.286471 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:02.286472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:02.287100 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:02.287103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:02.287509 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:02.287510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:02.287572 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:02.288123 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:02.288129 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:02.290226 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":99,"threads_started":1}
W20260928 14:54:02.290441 30157 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:02.285114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:49987 (local address 127.20.116.190:34953)
0928 14:54:02.285189 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:02.285192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:02.285305 (+ 113us) server_negotiation.cc:407] Connection header received
0928 14:54:02.285412 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:02.285413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:02.285438 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:02.285508 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:02.285855 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:02.286370 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:02.287214 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:02.287416 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:02.287505 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:02.290338 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.131:49987: BlockingRecv error: recv got EOF from 127.20.116.131:49987 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 14:54:03 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:03 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:03.011222 30160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:03.006150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.129:56229)
0928 14:54:03.006297 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:03.006309 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:03.006371 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:03.006567 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:03.006568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:03.006743 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:03.006794 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:03.006795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:03.007477 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:03.007478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:03.008014 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:03.008016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:03.008425 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:03.008427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:03.008489 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:03.009048 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:03.009055 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:03.011141 (+ 2086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
W20260928 14:54:03.011402 30161 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:03.006292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:56229 (local address 127.20.116.190:34953)
0928 14:54:03.006421 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:03.006423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:03.006439 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:54:03.006461 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:03.006462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:03.006474 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:03.006562 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:03.006873 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:03.007389 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:03.008118 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:03.008325 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:03.008390 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:03.011231 (+ 2841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.129:56229: BlockingRecv error: recv got EOF from 127.20.116.129:56229 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
Sep 28 14:54:03 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:03 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:03.144411 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:03.139645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.130:44391)
0928 14:54:03.139856 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:03.139868 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:03.139972 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:03.140127 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:03.140128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:03.140325 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:03.140374 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:03.140376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:03.141061 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:03.141063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:03.141590 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:03.141591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:03.141981 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:03.141982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:03.142045 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:03.142607 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:03.142612 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:03.144353 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":97,"threads_started":1}
W20260928 14:54:03.144531 30161 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:03.139748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:44391 (local address 127.20.116.190:34953)
0928 14:54:03.139831 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:03.139833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:03.139953 (+ 120us) server_negotiation.cc:407] Connection header received
0928 14:54:03.140034 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:03.140035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:03.140051 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:03.140122 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:03.140460 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:03.140969 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:03.141686 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:03.141904 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:03.141983 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:03.144451 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.130:44391: BlockingRecv error: recv got EOF from 127.20.116.130:44391 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 14:54:03 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:03 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:03.295814 30163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:03.290812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.131:39241)
0928 14:54:03.290973 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:03.290983 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:03.291033 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:03.291263 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:03.291264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:03.291447 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:03.291494 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:03.291496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:03.292217 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:03.292218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:03.292806 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:03.292808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:03.293254 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:03.293256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:03.293325 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:03.293917 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:03.293922 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:03.295749 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
W20260928 14:54:03.295966 30161 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:03.291045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:39241 (local address 127.20.116.190:34953)
0928 14:54:03.291124 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:03.291127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:03.291134 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:54:03.291154 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:03.291155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:03.291184 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:03.291253 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:03.291578 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:03.292106 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:03.292891 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:03.293182 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:03.293253 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:03.295869 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.131:39241: BlockingRecv error: recv got EOF from 127.20.116.131:39241 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 14:54:04 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:04 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:04.017150 30164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:04.011841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.129:58893)
0928 14:54:04.012072 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:04.012089 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:04.012186 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:04.012342 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:04.012343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:04.012505 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:04.012560 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:04.012562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:04.013312 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:04.013313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:04.013831 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:04.013833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:04.014220 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:04.014220 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:04.014263 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:04.014839 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:04.014844 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:04.017072 (+ 2228us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":73,"threads_started":1}
W20260928 14:54:04.017300 30165 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:04.011883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:58893 (local address 127.20.116.190:34953)
0928 14:54:04.011994 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:04.011997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:04.012195 (+ 198us) server_negotiation.cc:407] Connection header received
0928 14:54:04.012235 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:04.012236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:04.012251 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:04.012322 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:04.012668 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:04.013235 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:04.013914 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:04.014141 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:04.014205 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:04.017180 (+ 2975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.129:58893: BlockingRecv error: recv got EOF from 127.20.116.129:58893 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":44,"threads_started":1}
Sep 28 14:54:04 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:04 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:04.149899 30166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:04.144946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.130:36431)
0928 14:54:04.145166 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:04.145178 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:04.145278 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:04.145451 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:04.145452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:04.145641 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:04.145692 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:04.145694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:04.146429 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:04.146430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:04.147038 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:04.147040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:04.147441 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:04.147442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:04.147498 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:04.147959 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:04.147964 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:04.149827 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":100,"threads_started":1}
W20260928 14:54:04.149974 30165 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:04.145051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:36431 (local address 127.20.116.190:34953)
0928 14:54:04.145162 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:04.145164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:04.145256 (+ 92us) server_negotiation.cc:407] Connection header received
0928 14:54:04.145353 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:04.145354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:04.145374 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:04.145442 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:04.145814 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:04.146337 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:04.147138 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:04.147354 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:04.147442 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:04.149925 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.130:36431: BlockingRecv error: recv got EOF from 127.20.116.130:36431 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 14:54:04 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:04 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:04.301261 30167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:04.296389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.131:40237)
0928 14:54:04.296548 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:04.296559 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:04.296613 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:04.296759 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:04.296760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:04.296962 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:04.297015 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:04.297017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:04.297793 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:04.297794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:04.298369 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:04.298370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:04.298795 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:04.298796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:04.298851 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:04.299338 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:04.299343 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:04.301189 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
W20260928 14:54:04.301380 30165 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:04.296502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:40237 (local address 127.20.116.190:34953)
0928 14:54:04.296597 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:04.296599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:04.296617 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:54:04.296661 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:04.296661 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:04.296676 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:04.296742 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:04.297100 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:04.297705 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:04.298469 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:04.298689 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:04.298790 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:04.301300 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.131:40237: BlockingRecv error: recv got EOF from 127.20.116.131:40237 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 14:54:05 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:05 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:05.022226 30168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:05.017625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.129:51663)
0928 14:54:05.017811 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:05.017823 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:05.017874 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:05.018011 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:05.018012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:05.018174 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:05.018226 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:05.018228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:05.018934 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:05.018935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:05.019483 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:05.019485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:05.019818 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:05.019819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:05.019860 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:05.020369 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:05.020380 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:05.022152 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":119,"threads_started":1}
W20260928 14:54:05.022373 30169 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:05.017731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:51663 (local address 127.20.116.190:34953)
0928 14:54:05.017877 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:05.017879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:05.017908 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:54:05.017929 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:05.017929 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:05.017941 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:05.017992 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:05.018299 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:05.018868 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:05.019559 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:05.019753 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:05.019803 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:05.022278 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.129:51663: BlockingRecv error: recv got EOF from 127.20.116.129:51663 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
Sep 28 14:54:05 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:05 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:05.155381 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:05.150473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.130:57833)
0928 14:54:05.150736 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:05.150756 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:05.150822 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:05.150964 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:05.150965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:05.151140 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:05.151207 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:05.151208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:05.151928 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:05.151929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:05.152482 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:05.152484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:05.152885 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:05.152886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:05.152938 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:05.153399 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:05.153404 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:05.155320 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":100,"threads_started":1}
W20260928 14:54:05.155514 30169 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:05.150700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:57833 (local address 127.20.116.190:34953)
0928 14:54:05.150775 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:05.150777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:05.150805 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:54:05.150883 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:05.150884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:05.150896 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:05.150943 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:05.151277 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:05.151844 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:05.152578 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:05.152787 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:05.152852 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:05.155431 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.130:57833: BlockingRecv error: recv got EOF from 127.20.116.130:57833 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 14:54:05 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:05 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:05.306829 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:05.301878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.131:52383)
0928 14:54:05.302069 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:05.302081 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:05.302157 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:05.302316 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:05.302317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:05.302592 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:05.302651 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:05.302653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:05.303439 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:05.303440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:05.304011 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:05.304013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:05.304394 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:05.304395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:05.304455 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:05.304905 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:05.304911 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:05.306769 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":92,"threads_started":1}
W20260928 14:54:05.306967 30169 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:05.301987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:52383 (local address 127.20.116.190:34953)
0928 14:54:05.302062 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:05.302064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:05.302134 (+ 70us) server_negotiation.cc:407] Connection header received
0928 14:54:05.302215 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:05.302216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:05.302232 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:05.302308 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:05.302746 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:05.303361 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:05.304102 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:05.304308 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:05.304393 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:05.306882 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.131:52383: BlockingRecv error: recv got EOF from 127.20.116.131:52383 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 14:54:06 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:06 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:06.027863 30172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:06.022739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.129:53369)
0928 14:54:06.022954 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:06.022967 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:06.023041 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:06.023271 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:06.023273 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:06.023459 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:06.023515 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:06.023517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:06.024222 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:06.024223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:06.024778 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:06.024780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:06.025158 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:06.025158 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:06.025198 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:06.025669 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:06.025674 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:06.027759 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":93,"threads_started":1}
W20260928 14:54:06.027918 30173 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:06.022848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:53369 (local address 127.20.116.190:34953)
0928 14:54:06.022967 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:06.022969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:06.023034 (+ 65us) server_negotiation.cc:407] Connection header received
0928 14:54:06.023138 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:06.023139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:06.023154 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:06.023226 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:06.023615 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:06.024143 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:06.024865 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:06.025069 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:06.025144 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:06.027845 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.129:53369: BlockingRecv error: recv got EOF from 127.20.116.129:53369 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":53,"threads_started":1}
Sep 28 14:54:06 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:06 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:06.161052 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:06.155970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.130:53595)
0928 14:54:06.156154 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:06.156168 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:06.156236 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:06.156399 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:06.156400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:06.156607 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:06.156657 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:06.156659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:06.157401 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:06.157402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:06.157959 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:06.157961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:06.158350 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:06.158351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:06.158411 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:06.158920 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:06.158925 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:06.160988 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
W20260928 14:54:06.161168 30173 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:06.156118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:53595 (local address 127.20.116.190:34953)
0928 14:54:06.156197 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:06.156199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:06.156215 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:54:06.156323 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:06.156324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:06.156338 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:06.156386 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:06.156745 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:06.157303 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:06.158055 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:06.158270 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:06.158350 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:06.161091 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.130:53595: BlockingRecv error: recv got EOF from 127.20.116.130:53595 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 14:54:06 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:06 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:06.312744 30175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:06.307394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.131:36097)
0928 14:54:06.307541 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:06.307552 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:06.307626 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:06.307795 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:06.307796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:06.307983 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:06.308032 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:06.308034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:06.308812 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:06.308813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:06.309383 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:06.309385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:06.309799 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:06.309800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:06.309846 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:06.310399 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:06.310405 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:06.312631 (+ 2226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
W20260928 14:54:06.312789 30173 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:06.307512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:36097 (local address 127.20.116.190:34953)
0928 14:54:06.307563 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:06.307565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:06.307626 (+ 61us) server_negotiation.cc:407] Connection header received
0928 14:54:06.307686 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:06.307687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:06.307700 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:06.307744 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:06.308170 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:06.308713 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:06.309509 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:06.309721 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:06.309789 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:06.312741 (+ 2952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.131:36097: BlockingRecv error: recv got EOF from 127.20.116.131:36097 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 14:54:07 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:07 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:07.038514 30176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:07.028450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.129:33131)
0928 14:54:07.028610 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:07.028622 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:07.028692 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:07.028856 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:07.028857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:07.029029 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:07.029084 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:07.029085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:07.029842 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:07.029843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:07.030687 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:07.030690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:07.031083 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:07.031084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:07.031142 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:07.036339 (+ 5197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:07.036344 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:07.038403 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
W20260928 14:54:07.038609 30177 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:07.028546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:33131 (local address 127.20.116.190:34953)
0928 14:54:07.028687 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:07.028689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:07.028737 (+ 48us) server_negotiation.cc:407] Connection header received
0928 14:54:07.028764 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:07.028765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:07.028777 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:07.028832 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:07.029158 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:07.029756 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:07.030785 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:07.030992 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:07.031087 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:07.038505 (+ 7418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.129:33131: BlockingRecv error: recv got EOF from 127.20.116.129:33131 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
Sep 28 14:54:07 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:07 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:07.166744 30178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:07.161637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.130:35723)
0928 14:54:07.161872 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:07.161885 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:07.161983 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:07.162184 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:07.162185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:07.162440 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:07.162504 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:07.162507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:07.163222 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:07.163222 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:07.163821 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:07.163823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:07.164194 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:07.164195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:07.164248 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:07.164854 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:07.164867 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:07.166668 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":134,"threads_started":1}
W20260928 14:54:07.166867 30177 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:07.161822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:35723 (local address 127.20.116.190:34953)
0928 14:54:07.161887 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:07.161893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:07.161979 (+ 86us) server_negotiation.cc:407] Connection header received
0928 14:54:07.162051 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:07.162052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:07.162073 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:07.162145 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:07.162601 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:07.163124 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:07.163909 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:07.164102 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:07.164160 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:07.166790 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.130:35723: BlockingRecv error: recv got EOF from 127.20.116.130:35723 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 14:54:07 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:07 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.131@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:07.318651 30179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:07.313393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.131:36941)
0928 14:54:07.313593 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:07.313616 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:07.313705 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:07.313857 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:07.313858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:07.314042 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:07.314100 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:07.314102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:07.314903 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:07.314904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:07.315506 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:07.315507 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:07.315877 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:07.315878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:07.315930 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:07.316434 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:07.316441 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:07.318517 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":81,"threads_started":1}
W20260928 14:54:07.318706 30177 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:07.313537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:36941 (local address 127.20.116.190:34953)
0928 14:54:07.313628 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:07.313630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:07.313676 (+ 46us) server_negotiation.cc:407] Connection header received
0928 14:54:07.313760 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:07.313761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:07.313776 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:07.313829 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:07.314223 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:07.314801 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:07.315593 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:07.315799 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:07.315858 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:07.318646 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.131:36941: BlockingRecv error: recv got EOF from 127.20.116.131:36941 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 14:54:08 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:08 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.129@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:08.043880 30180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:08.039198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.129:42999)
0928 14:54:08.039353 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:08.039370 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:08.039423 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:08.039583 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:08.039584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:08.039748 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:08.039799 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:08.039801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:08.040461 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:08.040462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:08.041002 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:08.041003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:08.041382 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:08.041382 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:08.041426 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:08.041902 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:08.041907 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:08.043779 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
W20260928 14:54:08.043964 30181 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:08.039270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:42999 (local address 127.20.116.190:34953)
0928 14:54:08.039411 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:08.039413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:08.039422 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:54:08.039471 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:08.039471 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:08.039483 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:08.039547 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:08.039877 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:08.040379 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:08.041114 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:08.041310 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:08.041369 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:08.043868 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.129:42999: BlockingRecv error: recv got EOF from 127.20.116.129:42999 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
Sep 28 14:54:08 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:54:08 dist-test-slave-pxl6 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.116.130@KRBTEST.COM for oryx/127.20.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:54:08.172127 30182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:08.167361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34953 (local address 127.20.116.130:36051)
0928 14:54:08.167534 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:08.167546 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:08.167652 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:08.167788 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:08.167789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:08.167951 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:08.168002 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:08.168003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:08.168659 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:08.168659 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:08.169193 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:08.169195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:08.169604 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:08.169605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:08.169654 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:08.170121 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:08.170126 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:08.172056 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.116.190:34953: Server oryx/127.20.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":103,"threads_started":1}
W20260928 14:54:08.172260 30181 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:08.167436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:36051 (local address 127.20.116.190:34953)
0928 14:54:08.167511 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:08.167513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:08.167650 (+ 137us) server_negotiation.cc:407] Connection header received
0928 14:54:08.167712 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:08.167713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:08.167725 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:08.167767 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:08.168063 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:08.168586 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:08.169287 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:08.169484 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:08.169538 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:08.172176 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.116.130:36051: BlockingRecv error: recv got EOF from 127.20.116.130:36051 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:54:08.231231 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 29704
I20260928 14:54:08.236392 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 29856
I20260928 14:54:08.241029 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 29989
I20260928 14:54:08.245657 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 29634
2026-09-28T14:54:08Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17601 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T14:54:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:54:08Z Disabled control of system clock
I20260928 14:54:08.274725 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:34179
--webserver_interface=127.20.116.190
--webserver_port=0
--builtin_ntp_servers=127.20.116.148:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:34179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 14:54:08.356904 30186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:54:08.357117 30186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:54:08.357138 30186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:54:08.358265 30186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:54:08.358301 30186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:54:08.358314 30186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:54:08.358327 30186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:54:08.358340 30186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:54:08.359752 30186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:34179
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:34179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:54:08.359972 30186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:54:08.360230 30186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:54:08.365630 30186 server_base.cc:1034] running on GCE node
W20260928 14:54:08.365760 30191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:08.365754 30192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:08.365754 30194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:08.366130 30186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:54:08.366396 30186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:54:08.367565 30186 hybrid_clock.cc:648] HybridClock initialized: now 1790607248367544 us; error 34 us; skew 500 ppm
I20260928 14:54:08.368940 30186 webserver.cc:460] Webserver started at http://127.20.116.190:44781/ using document root <none> and password file <none>
I20260928 14:54:08.369222 30186 fs_manager.cc:359] Metadata directory not provided
I20260928 14:54:08.369271 30186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:54:08.369427 30186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:54:08.370573 30186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "be5f2bc19cc943ee95f41a6a7b6dc926"
format_stamp: "Formatted at 2026-09-28 14:54:08 on dist-test-slave-pxl6"
server_key: "382a9c6450e62c64fe127fab7b6c2f8f"
server_key_iv: "452e67b13b246aec3e38d6b6bb11f4ab"
server_key_version: "encryptionkey@0"
I20260928 14:54:08.370967 30186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "be5f2bc19cc943ee95f41a6a7b6dc926"
format_stamp: "Formatted at 2026-09-28 14:54:08 on dist-test-slave-pxl6"
server_key: "382a9c6450e62c64fe127fab7b6c2f8f"
server_key_iv: "452e67b13b246aec3e38d6b6bb11f4ab"
server_key_version: "encryptionkey@0"
I20260928 14:54:08.372313 30186 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:54:08.373081 30200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:08.373282 30186 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:54:08.373394 30186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "be5f2bc19cc943ee95f41a6a7b6dc926"
format_stamp: "Formatted at 2026-09-28 14:54:08 on dist-test-slave-pxl6"
server_key: "382a9c6450e62c64fe127fab7b6c2f8f"
server_key_iv: "452e67b13b246aec3e38d6b6bb11f4ab"
server_key_version: "encryptionkey@0"
I20260928 14:54:08.373473 30186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:54:08.388612 30186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:54:08.389001 30186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:54:08.389163 30186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:54:08.393466 30186 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:34179
I20260928 14:54:08.393503 30252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:34179 every 8 connection(s)
I20260928 14:54:08.393865 30186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:54:08.394690 30253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:54:08.397437 30253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926: Bootstrap starting.
I20260928 14:54:08.398064 30253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926: Neither blocks nor log segments found. Creating new log.
I20260928 14:54:08.398375 30253 log.cc:824] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926: Log is configured to *not* fsync() on all Append() calls
I20260928 14:54:08.399124 30253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926: No bootstrap required, opened a new log
I20260928 14:54:08.399188 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 30186
I20260928 14:54:08.399295 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:54:08.399485 20946 external_mini_cluster.cc:1428] Setting key 1200b64e7acc064ed4385581514605a5
I20260928 14:54:08.401183 30253 raft_consensus.cc:348] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5f2bc19cc943ee95f41a6a7b6dc926" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34179 } }
I20260928 14:54:08.401363 30253 raft_consensus.cc:374] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:54:08.401403 30253 raft_consensus.cc:729] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be5f2bc19cc943ee95f41a6a7b6dc926, State: Initialized, Role: FOLLOWER
I20260928 14:54:08.401577 30253 consensus_queue.cc:260] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5f2bc19cc943ee95f41a6a7b6dc926" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34179 } }
I20260928 14:54:08.401681 30253 raft_consensus.cc:388] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:54:08.401731 30253 raft_consensus.cc:482] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:54:08.401782 30253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:54:08.402357 30253 raft_consensus.cc:504] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5f2bc19cc943ee95f41a6a7b6dc926" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34179 } }
I20260928 14:54:08.402447 30253 leader_election.cc:302] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be5f2bc19cc943ee95f41a6a7b6dc926; no voters:
I20260928 14:54:08.402675 30253 leader_election.cc:288] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:54:08.402709 30258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:54:08.402871 30258 raft_consensus.cc:686] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926 [term 1 LEADER]: Becoming Leader. State: Replica: be5f2bc19cc943ee95f41a6a7b6dc926, State: Running, Role: LEADER
I20260928 14:54:08.402962 30253 sys_catalog.cc:564] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:54:08.403012 30258 consensus_queue.cc:237] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5f2bc19cc943ee95f41a6a7b6dc926" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34179 } }
I20260928 14:54:08.403431 30259 sys_catalog.cc:455] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be5f2bc19cc943ee95f41a6a7b6dc926" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5f2bc19cc943ee95f41a6a7b6dc926" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34179 } } }
I20260928 14:54:08.403523 30259 sys_catalog.cc:458] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926 [sys.catalog]: This master's current role is: LEADER
I20260928 14:54:08.403759 30260 sys_catalog.cc:455] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be5f2bc19cc943ee95f41a6a7b6dc926. Latest consensus state: current_term: 1 leader_uuid: "be5f2bc19cc943ee95f41a6a7b6dc926" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5f2bc19cc943ee95f41a6a7b6dc926" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34179 } } }
I20260928 14:54:08.403880 30260 sys_catalog.cc:458] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926 [sys.catalog]: This master's current role is: LEADER
I20260928 14:54:08.404137 30265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:54:08.404721 30265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:54:08.406606 30265 catalog_manager.cc:1347] Generated new cluster ID: 7040ed21e95148bf8bfb17ed646155a4
I20260928 14:54:08.406659 30265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:54:08.406664 30257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:08.400146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51940 (local address 127.20.116.190:34179)
0928 14:54:08.400336 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:08.400343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:08.400368 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:54:08.401161 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:08.401167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:08.401243 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:08.401381 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:08.401647 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:08.403207 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:08.404319 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:08.404940 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:08.405053 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:08.405861 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:54:08.405865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:54:08.405866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:54:08.406068 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:54:08.406111 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:54:08.406126 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:54:08.406197 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:08.406305 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 14:54:08.406396 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260928 14:54:08.416890 30265 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:54:08.417441 30265 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:54:08.421919 30265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be5f2bc19cc943ee95f41a6a7b6dc926: Generated new TSK 0
I20260928 14:54:08.422118 30265 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:54:08.425328 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34179
--builtin_ntp_servers=127.20.116.148:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 14:54:08.519999 30277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:54:08.520210 30277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:54:08.520232 30277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:54:08.521567 30277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:54:08.521615 30277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:54:08.521641 30277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:54:08.523062 30277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:54:08.523321 30277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:54:08.523653 30277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:54:08.528852 30283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:08.528846 30285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:08.528842 30282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:08.529256 30277 server_base.cc:1034] running on GCE node
I20260928 14:54:08.529470 30277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:54:08.529745 30277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:54:08.530927 30277 hybrid_clock.cc:648] HybridClock initialized: now 1790607248530914 us; error 32 us; skew 500 ppm
I20260928 14:54:08.532356 30277 webserver.cc:460] Webserver started at http://127.20.116.129:44887/ using document root <none> and password file <none>
I20260928 14:54:08.532619 30277 fs_manager.cc:359] Metadata directory not provided
I20260928 14:54:08.532675 30277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:54:08.532797 30277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:54:08.534154 30277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "2431dc7ea93b4820baf242ecbf832d3d"
format_stamp: "Formatted at 2026-09-28 14:54:08 on dist-test-slave-pxl6"
server_key: "2a61161efc15d9a2c13ce2c0f920a63e"
server_key_iv: "1fadbca23a4e391f3268a23ccf75e168"
server_key_version: "encryptionkey@0"
I20260928 14:54:08.534570 30277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "2431dc7ea93b4820baf242ecbf832d3d"
format_stamp: "Formatted at 2026-09-28 14:54:08 on dist-test-slave-pxl6"
server_key: "2a61161efc15d9a2c13ce2c0f920a63e"
server_key_iv: "1fadbca23a4e391f3268a23ccf75e168"
server_key_version: "encryptionkey@0"
I20260928 14:54:08.536147 30277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:54:08.537053 30291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:08.537273 30277 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:54:08.537362 30277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "2431dc7ea93b4820baf242ecbf832d3d"
format_stamp: "Formatted at 2026-09-28 14:54:08 on dist-test-slave-pxl6"
server_key: "2a61161efc15d9a2c13ce2c0f920a63e"
server_key_iv: "1fadbca23a4e391f3268a23ccf75e168"
server_key_version: "encryptionkey@0"
I20260928 14:54:08.537432 30277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:54:08.562263 30277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:54:08.562633 30277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:54:08.562749 30277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:54:08.562964 30277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:54:08.563268 30277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:54:08.563298 30277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:08.563321 30277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:54:08.563333 30277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:08.569515 30277 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:43667
I20260928 14:54:08.569571 30404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:43667 every 8 connection(s)
I20260928 14:54:08.569914 30277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:54:08.574097 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 30277
I20260928 14:54:08.574241 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:54:08.574440 20946 external_mini_cluster.cc:1428] Setting key 004b3c34d63ff388eb16c8ead30a8c14
I20260928 14:54:08.575443 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34179
--builtin_ntp_servers=127.20.116.148:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 14:54:08.575651 30257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:08.570766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:33811 (local address 127.20.116.190:34179)
0928 14:54:08.570853 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:08.570855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:08.571062 (+ 207us) server_negotiation.cc:407] Connection header received
0928 14:54:08.571671 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:08.571672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:08.571687 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:08.571753 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:08.572417 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:08.573031 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:08.574065 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:08.574325 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:08.574390 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:08.575357 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:54:08.575359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:54:08.575359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:54:08.575365 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:54:08.575380 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:54:08.575384 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:54:08.575434 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:08.575540 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 14:54:08.575572 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:54:08.575897 30407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:08.570733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34179 (local address 127.20.116.129:33811)
0928 14:54:08.570933 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:08.570969 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:08.571478 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:08.571808 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:08.571812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:08.571821 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:08.572328 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:08.572333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:08.573122 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:08.573123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:08.573959 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:08.573962 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:08.574401 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:08.574402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:08.574562 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:08.575200 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:54:08.575201 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:08.575235 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:54:08.575260 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:54:08.575434 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:54:08.575439 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:54:08.575443 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:54:08.575528 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 14:54:08.575637 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":105,"threads_started":1}
I20260928 14:54:08.576499 30405 heartbeater.cc:347] Connected to a master server at 127.20.116.190:34179
I20260928 14:54:08.576622 30405 heartbeater.cc:464] Registering TS with master...
I20260928 14:54:08.576824 30405 heartbeater.cc:511] Master 127.20.116.190:34179 requested a full tablet report, sending...
I20260928 14:54:08.577479 30217 ts_manager.cc:194] Registered new tserver with Master: 2431dc7ea93b4820baf242ecbf832d3d (127.20.116.129:43667)
I20260928 14:54:08.578125 30217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.116.129:33811
W20260928 14:54:08.659824 30408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:54:08.660068 30408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:54:08.660091 30408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:54:08.661281 30408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:54:08.661327 30408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:54:08.661353 30408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:54:08.662775 30408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:54:08.663000 30408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:54:08.663268 30408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:54:08.668676 30416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:08.668725 30413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:08.668731 30414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:08.668814 30408 server_base.cc:1034] running on GCE node
I20260928 14:54:08.669090 30408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:54:08.669322 30408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:54:08.670519 30408 hybrid_clock.cc:648] HybridClock initialized: now 1790607248670490 us; error 60 us; skew 500 ppm
I20260928 14:54:08.672084 30408 webserver.cc:460] Webserver started at http://127.20.116.130:41683/ using document root <none> and password file <none>
I20260928 14:54:08.672356 30408 fs_manager.cc:359] Metadata directory not provided
I20260928 14:54:08.672428 30408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:54:08.672590 30408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:54:08.673764 30408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "58d26637de884fde8661b7242eaea123"
format_stamp: "Formatted at 2026-09-28 14:54:08 on dist-test-slave-pxl6"
server_key: "6e9f468eef36270d4456fa87d48a6444"
server_key_iv: "e30f88a02eceb9f401e65d6173af55a2"
server_key_version: "encryptionkey@0"
I20260928 14:54:08.674151 30408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "58d26637de884fde8661b7242eaea123"
format_stamp: "Formatted at 2026-09-28 14:54:08 on dist-test-slave-pxl6"
server_key: "6e9f468eef36270d4456fa87d48a6444"
server_key_iv: "e30f88a02eceb9f401e65d6173af55a2"
server_key_version: "encryptionkey@0"
I20260928 14:54:08.675696 30408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260928 14:54:08.676625 30422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:08.676813 30408 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:54:08.676909 30408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "58d26637de884fde8661b7242eaea123"
format_stamp: "Formatted at 2026-09-28 14:54:08 on dist-test-slave-pxl6"
server_key: "6e9f468eef36270d4456fa87d48a6444"
server_key_iv: "e30f88a02eceb9f401e65d6173af55a2"
server_key_version: "encryptionkey@0"
I20260928 14:54:08.676973 30408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:54:08.687553 30408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:54:08.687846 30408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:54:08.687948 30408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:54:08.688130 30408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:54:08.688480 30408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:54:08.688516 30408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:08.688539 30408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:54:08.688551 30408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:08.694799 30408 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:34055
I20260928 14:54:08.694859 30535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:34055 every 8 connection(s)
I20260928 14:54:08.695201 30408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:54:08.700829 30257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:08.696014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:38871 (local address 127.20.116.190:34179)
0928 14:54:08.696080 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:08.696083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:08.696341 (+ 258us) server_negotiation.cc:407] Connection header received
0928 14:54:08.696958 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:08.696959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:08.696972 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:08.697022 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:08.697748 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:08.698269 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:08.699309 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:08.699513 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:08.699590 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:08.700492 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:54:08.700494 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:54:08.700495 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:54:08.700500 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:54:08.700517 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:54:08.700523 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:54:08.700589 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:08.700706 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 14:54:08.700741 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 14:54:08.701071 30538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:08.695995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34179 (local address 127.20.116.130:38871)
0928 14:54:08.696212 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:08.696248 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:08.696834 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:08.697114 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:08.697118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:08.697127 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:08.697634 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:08.697638 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:08.698374 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:08.698375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:08.699204 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:08.699207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:08.699612 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:08.699613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:08.699755 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:08.700334 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:54:08.700335 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:08.700364 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:54:08.700389 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:54:08.700591 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:54:08.700598 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:54:08.700601 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:54:08.700706 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 14:54:08.700823 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
I20260928 14:54:08.701164 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 30408
I20260928 14:54:08.701295 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:54:08.701506 20946 external_mini_cluster.cc:1428] Setting key 44b56ca4c51c0d276e7cd0adfea04e6e
I20260928 14:54:08.701691 30536 heartbeater.cc:347] Connected to a master server at 127.20.116.190:34179
I20260928 14:54:08.701789 30536 heartbeater.cc:464] Registering TS with master...
I20260928 14:54:08.702018 30536 heartbeater.cc:511] Master 127.20.116.190:34179 requested a full tablet report, sending...
I20260928 14:54:08.702414 30217 ts_manager.cc:194] Registered new tserver with Master: 58d26637de884fde8661b7242eaea123 (127.20.116.130:34055)
I20260928 14:54:08.702530 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34179
--builtin_ntp_servers=127.20.116.148:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 14:54:08.702828 30217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.116.130:38871
W20260928 14:54:08.787778 30539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:54:08.787994 30539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:54:08.788017 30539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:54:08.789211 30539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:54:08.789263 30539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:54:08.789289 30539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:54:08.790681 30539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:54:08.790936 30539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:54:08.791255 30539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:54:08.796384 30547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:08.796384 30544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:08.796407 30545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:08.796680 30539 server_base.cc:1034] running on GCE node
I20260928 14:54:08.797499 30539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:54:08.797832 30539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:54:08.799005 30539 hybrid_clock.cc:648] HybridClock initialized: now 1790607248798991 us; error 33 us; skew 500 ppm
I20260928 14:54:08.800325 30539 webserver.cc:460] Webserver started at http://127.20.116.131:37773/ using document root <none> and password file <none>
I20260928 14:54:08.800626 30539 fs_manager.cc:359] Metadata directory not provided
I20260928 14:54:08.800699 30539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:54:08.800873 30539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:54:08.802071 30539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "1f2d3d9e5a01446caf66777104e5b529"
format_stamp: "Formatted at 2026-09-28 14:54:08 on dist-test-slave-pxl6"
server_key: "b20f847a02cca110de6a015b0b6cd385"
server_key_iv: "b557c54f29298819d64477218cbe9173"
server_key_version: "encryptionkey@0"
I20260928 14:54:08.802513 30539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "1f2d3d9e5a01446caf66777104e5b529"
format_stamp: "Formatted at 2026-09-28 14:54:08 on dist-test-slave-pxl6"
server_key: "b20f847a02cca110de6a015b0b6cd385"
server_key_iv: "b557c54f29298819d64477218cbe9173"
server_key_version: "encryptionkey@0"
I20260928 14:54:08.804131 30539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:54:08.805042 30553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:08.805236 30539 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:54:08.805342 30539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "1f2d3d9e5a01446caf66777104e5b529"
format_stamp: "Formatted at 2026-09-28 14:54:08 on dist-test-slave-pxl6"
server_key: "b20f847a02cca110de6a015b0b6cd385"
server_key_iv: "b557c54f29298819d64477218cbe9173"
server_key_version: "encryptionkey@0"
I20260928 14:54:08.805435 30539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:54:08.814221 30539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:54:08.814630 30539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:54:08.814792 30539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:54:08.815032 30539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:54:08.815369 30539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:54:08.815423 30539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:08.815456 30539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:54:08.815483 30539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:08.821540 30539 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:41065
I20260928 14:54:08.821619 30666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:41065 every 8 connection(s)
I20260928 14:54:08.821977 30539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:54:08.827258 30257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:08.822795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:40821 (local address 127.20.116.190:34179)
0928 14:54:08.822842 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:08.822844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:08.823120 (+ 276us) server_negotiation.cc:407] Connection header received
0928 14:54:08.823579 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:08.823580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:08.823591 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:08.823654 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:08.824346 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:08.824856 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:08.825899 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:08.826083 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:08.826135 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:08.827005 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:54:08.827006 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:54:08.827006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:54:08.827011 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:54:08.827024 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:54:08.827028 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:54:08.827072 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:08.827162 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 14:54:08.827197 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:54:08.827462 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:08.822764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34179 (local address 127.20.116.131:40821)
0928 14:54:08.822956 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:08.822998 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:08.823489 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:08.823730 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:08.823734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:08.823743 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:08.824259 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:08.824263 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:08.824969 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:08.824970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:08.825817 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:08.825819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:08.826166 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:08.826167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:08.826282 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:08.826866 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:54:08.826866 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:08.826896 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:54:08.826920 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:54:08.827088 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:54:08.827091 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:54:08.827094 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:54:08.827146 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 14:54:08.827230 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260928 14:54:08.827984 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 30539
I20260928 14:54:08.828082 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:54:08.828100 30667 heartbeater.cc:347] Connected to a master server at 127.20.116.190:34179
I20260928 14:54:08.828226 30667 heartbeater.cc:464] Registering TS with master...
I20260928 14:54:08.828322 20946 external_mini_cluster.cc:1428] Setting key 9825ae5028e68b3af4402b712146f9af
I20260928 14:54:08.828460 30667 heartbeater.cc:511] Master 127.20.116.190:34179 requested a full tablet report, sending...
I20260928 14:54:08.828905 30217 ts_manager.cc:194] Registered new tserver with Master: 1f2d3d9e5a01446caf66777104e5b529 (127.20.116.131:41065)
I20260928 14:54:08.829242 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:54:08.829393 30217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.116.131:40821
W20260928 14:54:08.831262 30257 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:08.830755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51950 (local address 127.20.116.190:34179)
0928 14:54:08.830814 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:08.830815 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:08.830932 (+ 117us) server_negotiation.cc:407] Connection header received
0928 14:54:08.831013 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:08.831014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:08.831026 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:08.831075 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:08.831201 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51950: BlockingRecv error: recv got EOF from 127.0.0.1:51950 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:54:08.831413 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 30277
I20260928 14:54:08.837145 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 30408
I20260928 14:54:08.842092 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 30539
I20260928 14:54:08.846704 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 30186
2026-09-28T14:54:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (599 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T14:54:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:54:08Z Disabled control of system clock
I20260928 14:54:08.874253 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:46811
--webserver_interface=127.20.116.190
--webserver_port=0
--builtin_ntp_servers=127.20.116.148:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:46811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 14:54:08.958343 30678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:54:08.958587 30678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:54:08.958611 30678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:54:08.959762 30678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:54:08.959800 30678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:54:08.959813 30678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:54:08.959826 30678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:54:08.959837 30678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:54:08.961130 30678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:46811
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:46811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:54:08.961346 30678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:54:08.961592 30678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:54:08.966518 30678 server_base.cc:1034] running on GCE node
W20260928 14:54:08.966501 30684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:08.966490 30683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:08.966500 30686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:08.967061 30678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:54:08.967386 30678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:54:08.968575 30678 hybrid_clock.cc:648] HybridClock initialized: now 1790607248968548 us; error 38 us; skew 500 ppm
I20260928 14:54:08.969947 30678 webserver.cc:460] Webserver started at http://127.20.116.190:38631/ using document root <none> and password file <none>
I20260928 14:54:08.970199 30678 fs_manager.cc:359] Metadata directory not provided
I20260928 14:54:08.970296 30678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:54:08.970450 30678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:54:08.971627 30678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "a2612a3322ec4b14962be2b7accefc0e"
format_stamp: "Formatted at 2026-09-28 14:54:08 on dist-test-slave-pxl6"
server_key: "182760a808217fc571977d2df9de3ac7"
server_key_iv: "535950ff20b789eb0ff1cc03ca4d2df5"
server_key_version: "encryptionkey@0"
I20260928 14:54:08.972014 30678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "a2612a3322ec4b14962be2b7accefc0e"
format_stamp: "Formatted at 2026-09-28 14:54:08 on dist-test-slave-pxl6"
server_key: "182760a808217fc571977d2df9de3ac7"
server_key_iv: "535950ff20b789eb0ff1cc03ca4d2df5"
server_key_version: "encryptionkey@0"
I20260928 14:54:08.973404 30678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 14:54:08.974169 30692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:08.974414 30678 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:54:08.974666 30678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "a2612a3322ec4b14962be2b7accefc0e"
format_stamp: "Formatted at 2026-09-28 14:54:08 on dist-test-slave-pxl6"
server_key: "182760a808217fc571977d2df9de3ac7"
server_key_iv: "535950ff20b789eb0ff1cc03ca4d2df5"
server_key_version: "encryptionkey@0"
I20260928 14:54:08.974778 30678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:54:08.978232 30678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:54:08.978597 30678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:54:08.978756 30678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:54:08.982807 30678 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:46811
I20260928 14:54:08.982841 30744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:46811 every 8 connection(s)
I20260928 14:54:08.983194 30678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:54:08.983875 30745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:54:08.986693 30745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e: Bootstrap starting.
I20260928 14:54:08.987247 30745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e: Neither blocks nor log segments found. Creating new log.
I20260928 14:54:08.987493 30745 log.cc:824] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e: Log is configured to *not* fsync() on all Append() calls
I20260928 14:54:08.988157 30745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e: No bootstrap required, opened a new log
I20260928 14:54:08.988668 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 30678
I20260928 14:54:08.988881 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:54:08.989153 20946 external_mini_cluster.cc:1428] Setting key 320d4a82220b55ef5bbd5707d3f410ed
I20260928 14:54:08.990286 30745 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2612a3322ec4b14962be2b7accefc0e" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 46811 } }
I20260928 14:54:08.990479 30745 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:54:08.990522 30745 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2612a3322ec4b14962be2b7accefc0e, State: Initialized, Role: FOLLOWER
I20260928 14:54:08.990726 30745 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2612a3322ec4b14962be2b7accefc0e" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 46811 } }
I20260928 14:54:08.990823 30745 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:54:08.990902 30745 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:54:08.990968 30745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:54:08.991798 30745 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2612a3322ec4b14962be2b7accefc0e" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 46811 } }
I20260928 14:54:08.991892 30745 leader_election.cc:302] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2612a3322ec4b14962be2b7accefc0e; no voters:
I20260928 14:54:08.991912 30749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:08.989739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42358 (local address 127.20.116.190:46811)
0928 14:54:08.989958 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:08.989966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:08.989993 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:54:08.990257 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:08.990264 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:08.990304 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:08.990424 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:08.991355 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:54:08.991358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:54:08.991359 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:54:08.991363 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:54:08.991404 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:54:08.991417 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:54:08.991483 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:08.991574 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 14:54:08.991632 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20260928 14:54:08.992084 30745 leader_election.cc:288] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:54:08.992347 30745 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:54:08.992386 30750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:54:08.992553 30750 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e [term 1 LEADER]: Becoming Leader. State: Replica: a2612a3322ec4b14962be2b7accefc0e, State: Running, Role: LEADER
I20260928 14:54:08.992724 30750 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2612a3322ec4b14962be2b7accefc0e" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 46811 } }
I20260928 14:54:08.993271 30752 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2612a3322ec4b14962be2b7accefc0e. Latest consensus state: current_term: 1 leader_uuid: "a2612a3322ec4b14962be2b7accefc0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2612a3322ec4b14962be2b7accefc0e" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 46811 } } }
I20260928 14:54:08.993364 30752 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e [sys.catalog]: This master's current role is: LEADER
I20260928 14:54:08.993424 30751 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2612a3322ec4b14962be2b7accefc0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2612a3322ec4b14962be2b7accefc0e" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 46811 } } }
I20260928 14:54:08.993512 30751 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e [sys.catalog]: This master's current role is: LEADER
I20260928 14:54:08.993739 30762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:54:08.994362 30762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:54:08.995985 30762 catalog_manager.cc:1347] Generated new cluster ID: 53b2823622cb47e7a191543ca0406588
I20260928 14:54:08.996042 30762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:54:09.001391 30762 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:54:09.001580 30762 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:54:09.007236 30762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2612a3322ec4b14962be2b7accefc0e: Generated new TSK 0
I20260928 14:54:09.007465 30762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:54:09.014523 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:46811
--builtin_ntp_servers=127.20.116.148:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 14:54:09.101859 30769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:54:09.102098 30769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:54:09.102128 30769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:54:09.103542 30769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:54:09.103600 30769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:54:09.103639 30769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:54:09.105140 30769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:46811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:54:09.105544 30769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:54:09.105916 30769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:54:09.112089 30775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:09.112098 30774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:09.112344 30769 server_base.cc:1034] running on GCE node
W20260928 14:54:09.112106 30777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:09.112732 30769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:54:09.112972 30769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:54:09.114162 30769 hybrid_clock.cc:648] HybridClock initialized: now 1790607249114137 us; error 42 us; skew 500 ppm
I20260928 14:54:09.115686 30769 webserver.cc:460] Webserver started at http://127.20.116.129:40495/ using document root <none> and password file <none>
I20260928 14:54:09.115986 30769 fs_manager.cc:359] Metadata directory not provided
I20260928 14:54:09.116060 30769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:54:09.116212 30769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:54:09.117340 30769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "bb38df9c5ae34f31b2c74d5b58a8c7e0"
format_stamp: "Formatted at 2026-09-28 14:54:09 on dist-test-slave-pxl6"
server_key: "a960f888a0a87c29f9e28958a6c6c849"
server_key_iv: "46f1f14acb225e533937df9ab8feb69f"
server_key_version: "encryptionkey@0"
I20260928 14:54:09.117734 30769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "bb38df9c5ae34f31b2c74d5b58a8c7e0"
format_stamp: "Formatted at 2026-09-28 14:54:09 on dist-test-slave-pxl6"
server_key: "a960f888a0a87c29f9e28958a6c6c849"
server_key_iv: "46f1f14acb225e533937df9ab8feb69f"
server_key_version: "encryptionkey@0"
I20260928 14:54:09.119248 30769 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:54:09.120132 30783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:09.120323 30769 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:54:09.120440 30769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "bb38df9c5ae34f31b2c74d5b58a8c7e0"
format_stamp: "Formatted at 2026-09-28 14:54:09 on dist-test-slave-pxl6"
server_key: "a960f888a0a87c29f9e28958a6c6c849"
server_key_iv: "46f1f14acb225e533937df9ab8feb69f"
server_key_version: "encryptionkey@0"
I20260928 14:54:09.120514 30769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:54:09.124058 30769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:54:09.124363 30769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:54:09.124512 30769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:54:09.124732 30769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:54:09.125061 30769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:54:09.125118 30769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:09.125159 30769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:54:09.125186 30769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:09.131060 30769 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:42295
I20260928 14:54:09.131134 30896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:42295 every 8 connection(s)
I20260928 14:54:09.131479 30769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:54:09.134759 30749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:09.132279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:49807 (local address 127.20.116.190:46811)
0928 14:54:09.132366 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:09.132369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:09.132593 (+ 224us) server_negotiation.cc:407] Connection header received
0928 14:54:09.133137 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:09.133138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:09.133146 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:09.133195 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:09.134429 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:54:09.134430 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:54:09.134431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:54:09.134436 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:54:09.134454 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:54:09.134460 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:54:09.134539 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:09.134660 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 14:54:09.134680 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:54:09.134990 30899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:09.132257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:46811 (local address 127.20.116.129:49807)
0928 14:54:09.132462 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:09.132498 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:09.133021 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:09.133276 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:09.133280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:09.133290 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:09.134247 (+ 957us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:54:09.134248 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:09.134278 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:54:09.134307 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:54:09.134556 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:54:09.134563 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:54:09.134566 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:54:09.134671 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 14:54:09.134737 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260928 14:54:09.135614 30897 heartbeater.cc:347] Connected to a master server at 127.20.116.190:46811
I20260928 14:54:09.135699 30897 heartbeater.cc:464] Registering TS with master...
I20260928 14:54:09.135874 30897 heartbeater.cc:511] Master 127.20.116.190:46811 requested a full tablet report, sending...
I20260928 14:54:09.136413 30709 ts_manager.cc:194] Registered new tserver with Master: bb38df9c5ae34f31b2c74d5b58a8c7e0 (127.20.116.129:42295)
I20260928 14:54:09.139842 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 30769
I20260928 14:54:09.139915 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:54:09.140069 20946 external_mini_cluster.cc:1428] Setting key 834ad2a28a825603d3c8a3728cece263
I20260928 14:54:09.140897 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:46811
--builtin_ntp_servers=127.20.116.148:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 14:54:09.224160 30900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:54:09.224426 30900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:54:09.224462 30900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:54:09.225652 30900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:54:09.225720 30900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:54:09.225764 30900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:54:09.227262 30900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:46811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:54:09.227557 30900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:54:09.227866 30900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:54:09.233392 30905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:09.233448 30900 server_base.cc:1034] running on GCE node
W20260928 14:54:09.233392 30906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:09.233408 30908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:09.233920 30900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:54:09.234189 30900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:54:09.235361 30900 hybrid_clock.cc:648] HybridClock initialized: now 1790607249235357 us; error 38 us; skew 500 ppm
I20260928 14:54:09.236706 30900 webserver.cc:460] Webserver started at http://127.20.116.130:35219/ using document root <none> and password file <none>
I20260928 14:54:09.236965 30900 fs_manager.cc:359] Metadata directory not provided
I20260928 14:54:09.237036 30900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:54:09.237190 30900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:54:09.238363 30900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "8c45cda2f66a4be99a999e813504cb9a"
format_stamp: "Formatted at 2026-09-28 14:54:09 on dist-test-slave-pxl6"
server_key: "90f52d50fdfaebe0125092ea9d4bec73"
server_key_iv: "9508469138494e4d0f4d2f939ca14eef"
server_key_version: "encryptionkey@0"
I20260928 14:54:09.238795 30900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "8c45cda2f66a4be99a999e813504cb9a"
format_stamp: "Formatted at 2026-09-28 14:54:09 on dist-test-slave-pxl6"
server_key: "90f52d50fdfaebe0125092ea9d4bec73"
server_key_iv: "9508469138494e4d0f4d2f939ca14eef"
server_key_version: "encryptionkey@0"
I20260928 14:54:09.240234 30900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:54:09.241071 30914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:09.241243 30900 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:54:09.241356 30900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "8c45cda2f66a4be99a999e813504cb9a"
format_stamp: "Formatted at 2026-09-28 14:54:09 on dist-test-slave-pxl6"
server_key: "90f52d50fdfaebe0125092ea9d4bec73"
server_key_iv: "9508469138494e4d0f4d2f939ca14eef"
server_key_version: "encryptionkey@0"
I20260928 14:54:09.241438 30900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:54:09.245186 30900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:54:09.245509 30900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:54:09.245663 30900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:54:09.245913 30900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:54:09.246263 30900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:54:09.246318 30900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:09.246366 30900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:54:09.246397 30900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:09.252455 30900 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:38619
I20260928 14:54:09.252501 31027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:38619 every 8 connection(s)
I20260928 14:54:09.252905 30900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:54:09.255657 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 30900
I20260928 14:54:09.255775 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:54:09.256016 20946 external_mini_cluster.cc:1428] Setting key badf077ad7d0c1ca387ab8c0b761c659
I20260928 14:54:09.256492 30749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:09.253700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:56217 (local address 127.20.116.190:46811)
0928 14:54:09.253776 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:09.253778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:09.254027 (+ 249us) server_negotiation.cc:407] Connection header received
0928 14:54:09.254581 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:09.254582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:09.254590 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:09.254651 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:09.255994 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:54:09.255996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:54:09.255996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:54:09.256001 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:54:09.256020 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:54:09.256026 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:54:09.256121 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:09.256401 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 14:54:09.256417 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:54:09.257180 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:09.253696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:46811 (local address 127.20.116.130:56217)
0928 14:54:09.253899 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:09.253937 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:09.254442 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:09.254735 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:09.254739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:09.254748 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:09.255778 (+ 1030us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:54:09.255779 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:09.255822 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:54:09.255863 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:54:09.256117 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:54:09.256124 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:54:09.256128 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:54:09.256869 (+ 741us) client_negotiation.cc:240] Negotiation successful
0928 14:54:09.256957 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260928 14:54:09.257339 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:46811
--builtin_ntp_servers=127.20.116.148:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 14:54:09.257722 31028 heartbeater.cc:347] Connected to a master server at 127.20.116.190:46811
I20260928 14:54:09.257835 31028 heartbeater.cc:464] Registering TS with master...
I20260928 14:54:09.257997 31028 heartbeater.cc:511] Master 127.20.116.190:46811 requested a full tablet report, sending...
I20260928 14:54:09.258389 30709 ts_manager.cc:194] Registered new tserver with Master: 8c45cda2f66a4be99a999e813504cb9a (127.20.116.130:38619)
W20260928 14:54:09.341012 31031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:54:09.341238 31031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:54:09.341264 31031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:54:09.342633 31031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:54:09.342675 31031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:54:09.342703 31031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:54:09.344090 31031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:46811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:54:09.344383 31031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:54:09.344671 31031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:54:09.349912 31036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:09.349916 31037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:09.350111 31031 server_base.cc:1034] running on GCE node
W20260928 14:54:09.349997 31039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:09.350409 31031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:54:09.350728 31031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:54:09.351876 31031 hybrid_clock.cc:648] HybridClock initialized: now 1790607249351860 us; error 37 us; skew 500 ppm
I20260928 14:54:09.353140 31031 webserver.cc:460] Webserver started at http://127.20.116.131:43959/ using document root <none> and password file <none>
I20260928 14:54:09.353398 31031 fs_manager.cc:359] Metadata directory not provided
I20260928 14:54:09.353466 31031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:54:09.353654 31031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:54:09.354830 31031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "11f5a4186d4a4e32a670c61b537eb5c8"
format_stamp: "Formatted at 2026-09-28 14:54:09 on dist-test-slave-pxl6"
server_key: "3d1af307a65b09374269d4918ded3740"
server_key_iv: "543deebd717babfbe01e6324b98afb17"
server_key_version: "encryptionkey@0"
I20260928 14:54:09.355178 31031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "11f5a4186d4a4e32a670c61b537eb5c8"
format_stamp: "Formatted at 2026-09-28 14:54:09 on dist-test-slave-pxl6"
server_key: "3d1af307a65b09374269d4918ded3740"
server_key_iv: "543deebd717babfbe01e6324b98afb17"
server_key_version: "encryptionkey@0"
I20260928 14:54:09.356511 31031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:54:09.357407 31045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:09.357581 31031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:54:09.357694 31031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "11f5a4186d4a4e32a670c61b537eb5c8"
format_stamp: "Formatted at 2026-09-28 14:54:09 on dist-test-slave-pxl6"
server_key: "3d1af307a65b09374269d4918ded3740"
server_key_iv: "543deebd717babfbe01e6324b98afb17"
server_key_version: "encryptionkey@0"
I20260928 14:54:09.357776 31031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:54:09.361382 31031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:54:09.361666 31031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:54:09.361815 31031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:54:09.362027 31031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:54:09.362352 31031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:54:09.362403 31031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:09.362445 31031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:54:09.362471 31031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:09.368284 31031 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:36199
I20260928 14:54:09.368359 31158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:36199 every 8 connection(s)
I20260928 14:54:09.368712 31031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:54:09.371804 30749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:09.369440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:59687 (local address 127.20.116.190:46811)
0928 14:54:09.369507 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:09.369510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:09.369776 (+ 266us) server_negotiation.cc:407] Connection header received
0928 14:54:09.370271 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:09.370272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:09.370278 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:09.370331 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:09.371517 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:54:09.371518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:54:09.371519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:54:09.371523 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:54:09.371539 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:54:09.371543 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:54:09.371597 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:09.371727 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 14:54:09.371746 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 14:54:09.372011 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:09.369443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:46811 (local address 127.20.116.131:59687)
0928 14:54:09.369651 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:09.369686 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:09.370182 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:09.370386 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:09.370389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:09.370398 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:09.371362 (+ 964us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:54:09.371362 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:09.371392 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:54:09.371421 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:54:09.371631 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:54:09.371637 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:54:09.371641 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:54:09.371720 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 14:54:09.371779 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260928 14:54:09.372517 31159 heartbeater.cc:347] Connected to a master server at 127.20.116.190:46811
I20260928 14:54:09.372594 31159 heartbeater.cc:464] Registering TS with master...
I20260928 14:54:09.372714 31159 heartbeater.cc:511] Master 127.20.116.190:46811 requested a full tablet report, sending...
I20260928 14:54:09.372931 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 31031
I20260928 14:54:09.373013 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:54:09.373162 30709 ts_manager.cc:194] Registered new tserver with Master: 11f5a4186d4a4e32a670c61b537eb5c8 (127.20.116.131:36199)
I20260928 14:54:09.373260 20946 external_mini_cluster.cc:1428] Setting key 1730d92d8c71231d6843febba7c71d6a
I20260928 14:54:09.374133 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 14:54:09.376094 30749 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:54:09.375505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42366 (local address 127.20.116.190:46811)
0928 14:54:09.375579 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:09.375581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:09.375694 (+ 113us) server_negotiation.cc:407] Connection header received
0928 14:54:09.375825 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:09.375826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:09.375832 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:09.375895 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:09.376028 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42366: BlockingRecv error: recv got EOF from 127.0.0.1:42366 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:54:09.376150 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 30769
I20260928 14:54:09.381145 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 30900
I20260928 14:54:09.385491 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 31031
I20260928 14:54:09.390059 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 30678
2026-09-28T14:54:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (543 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:54:09 dist-test-slave-pxl6 krb5kdc[31169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:54:09 dist-test-slave-pxl6 krb5kdc[31169](info): set up 2 sockets
Sep 28 14:54:09 dist-test-slave-pxl6 krb5kdc[31169](info): commencing operation
krb5kdc: starting...
W20260928 14:54:10.432641 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:54:10 dist-test-slave-pxl6 krb5kdc[31169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:54:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:54:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:54:10.554052 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:34643
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:39293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:34643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:54:10.637006 31182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:54:10.637233 31182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:54:10.637259 31182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:54:10.638636 31182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:54:10.638684 31182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:54:10.638701 31182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:54:10.638715 31182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:54:10.638727 31182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:54:10.640318 31182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:39293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:34643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:34643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:54:10.640599 31182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:54:10.640853 31182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:54:10.646059 31188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:10.646193 31182 server_base.cc:1034] running on GCE node
W20260928 14:54:10.646185 31187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:10.646287 31190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:10.646723 31182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:54:10.647075 31182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:54:10.648293 31182 hybrid_clock.cc:648] HybridClock initialized: now 1790607250648273 us; error 31 us; skew 500 ppm
Sep 28 14:54:10 dist-test-slave-pxl6 krb5kdc[31169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607250, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:54:10.651010 31182 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:54:10.651403 31182 webserver.cc:460] Webserver started at http://127.20.116.190:43969/ using document root <none> and password file <none>
I20260928 14:54:10.651611 31182 fs_manager.cc:359] Metadata directory not provided
I20260928 14:54:10.651654 31182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:54:10.651743 31182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:54:10.652765 31182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "46830767e2ed4029af9f0aa529a5b1e2"
format_stamp: "Formatted at 2026-09-28 14:54:10 on dist-test-slave-pxl6"
server_key: "4e3f986b07822f3be0caeb531d8b8289"
server_key_iv: "46cb8be482ab78b94a40a84cf371f58f"
server_key_version: "encryptionkey@0"
I20260928 14:54:10.653055 31182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "46830767e2ed4029af9f0aa529a5b1e2"
format_stamp: "Formatted at 2026-09-28 14:54:10 on dist-test-slave-pxl6"
server_key: "4e3f986b07822f3be0caeb531d8b8289"
server_key_iv: "46cb8be482ab78b94a40a84cf371f58f"
server_key_version: "encryptionkey@0"
I20260928 14:54:10.654403 31182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:54:10.655242 31197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:10.655398 31182 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:54:10.655488 31182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "46830767e2ed4029af9f0aa529a5b1e2"
format_stamp: "Formatted at 2026-09-28 14:54:10 on dist-test-slave-pxl6"
server_key: "4e3f986b07822f3be0caeb531d8b8289"
server_key_iv: "46cb8be482ab78b94a40a84cf371f58f"
server_key_version: "encryptionkey@0"
I20260928 14:54:10.655545 31182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:54:10.669095 31182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:54:10.671074 31182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:54:10.671224 31182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:54:10.675235 31182 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:34643
I20260928 14:54:10.675288 31249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:34643 every 8 connection(s)
I20260928 14:54:10.675637 31182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:54:10.676712 31250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:54:10.679085 31250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2: Bootstrap starting.
I20260928 14:54:10.679101 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 31182
I20260928 14:54:10.679214 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:54:10.679411 20946 external_mini_cluster.cc:1428] Setting key 6415b2412da80511cae0c17937a1a8a3
I20260928 14:54:10.679847 31250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2: Neither blocks nor log segments found. Creating new log.
I20260928 14:54:10.680197 31250 log.cc:824] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2: Log is configured to *not* fsync() on all Append() calls
I20260928 14:54:10.680953 31250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2: No bootstrap required, opened a new log
I20260928 14:54:10.683197 31250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46830767e2ed4029af9f0aa529a5b1e2" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34643 } }
I20260928 14:54:10.683363 31250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:54:10.683393 31250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46830767e2ed4029af9f0aa529a5b1e2, State: Initialized, Role: FOLLOWER
I20260928 14:54:10.683509 31250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46830767e2ed4029af9f0aa529a5b1e2" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34643 } }
I20260928 14:54:10.683604 31250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:54:10.683645 31250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:54:10.683702 31250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:54:10.684268 31250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46830767e2ed4029af9f0aa529a5b1e2" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34643 } }
I20260928 14:54:10.684387 31250 leader_election.cc:302] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46830767e2ed4029af9f0aa529a5b1e2; no voters:
I20260928 14:54:10.684587 31250 leader_election.cc:288] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:54:10.684754 31256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:54:10.684880 31250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:54:10.684937 31256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2 [term 1 LEADER]: Becoming Leader. State: Replica: 46830767e2ed4029af9f0aa529a5b1e2, State: Running, Role: LEADER
I20260928 14:54:10.685030 31256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46830767e2ed4029af9f0aa529a5b1e2" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34643 } }
I20260928 14:54:10.685467 31256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46830767e2ed4029af9f0aa529a5b1e2. Latest consensus state: current_term: 1 leader_uuid: "46830767e2ed4029af9f0aa529a5b1e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46830767e2ed4029af9f0aa529a5b1e2" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34643 } } }
I20260928 14:54:10.685478 31255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46830767e2ed4029af9f0aa529a5b1e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46830767e2ed4029af9f0aa529a5b1e2" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34643 } } }
I20260928 14:54:10.685755 31256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2 [sys.catalog]: This master's current role is: LEADER
I20260928 14:54:10.685839 31255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2 [sys.catalog]: This master's current role is: LEADER
I20260928 14:54:10.686107 31265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 14:54:10 dist-test-slave-pxl6 krb5kdc[31169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:54:10.686764 31265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:54:10.688385 31265 catalog_manager.cc:1347] Generated new cluster ID: 17bcddddbe92451da754e8a4615505a9
I20260928 14:54:10.688439 31265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:54:10.692852 31253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:10.679975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45974 (local address 127.20.116.190:34643)
0928 14:54:10.680157 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:10.680166 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:10.680202 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:54:10.680671 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:10.680678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:10.680734 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:10.680849 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:10.681635 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:10.682509 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:10.683629 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:10.684097 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:10.684179 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:10.687226 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:54:10.687230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:54:10.687231 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:54:10.687242 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:54:10.691299 (+ 4057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:54:10.691716 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:54:10.691719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:54:10.691719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:54:10.691802 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:54:10.692019 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:54:10.692020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:54:10.692020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:54:10.692255 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:54:10.692437 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:10.692516 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 14:54:10.692658 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260928 14:54:10.731575 31265 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:54:10.732167 31265 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:54:10.734736 31265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2: Generated new TSK 0
I20260928 14:54:10.734930 31265 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
I20260928 14:54:10.765702 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34643
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:39293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:54:10.852823 31276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:54:10.853020 31276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:54:10.853044 31276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:54:10.854133 31276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:54:10.854168 31276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:54:10.854193 31276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:54:10.855657 31276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:39293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:54:10.855854 31276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:54:10.856068 31276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:54:10.861065 31284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:10.861109 31282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:10.861263 31276 server_base.cc:1034] running on GCE node
W20260928 14:54:10.861155 31281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:10.861528 31276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:54:10.861770 31276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:54:10.862954 31276 hybrid_clock.cc:648] HybridClock initialized: now 1790607250862941 us; error 27 us; skew 500 ppm
Sep 28 14:54:10 dist-test-slave-pxl6 krb5kdc[31169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607250, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:54:10.865499 31276 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:54:10.866010 31276 webserver.cc:460] Webserver started at http://127.20.116.129:33059/ using document root <none> and password file <none>
I20260928 14:54:10.866259 31276 fs_manager.cc:359] Metadata directory not provided
I20260928 14:54:10.866304 31276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:54:10.866464 31276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:54:10.867563 31276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "3bbeab0779cd4e518a6c333f5d7cb927"
format_stamp: "Formatted at 2026-09-28 14:54:10 on dist-test-slave-pxl6"
server_key: "056e0e5c9df5d279c29eb8656bbb31b1"
server_key_iv: "3b65fd830a07aa12f90a6d113bd2c267"
server_key_version: "encryptionkey@0"
I20260928 14:54:10.867928 31276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "3bbeab0779cd4e518a6c333f5d7cb927"
format_stamp: "Formatted at 2026-09-28 14:54:10 on dist-test-slave-pxl6"
server_key: "056e0e5c9df5d279c29eb8656bbb31b1"
server_key_iv: "3b65fd830a07aa12f90a6d113bd2c267"
server_key_version: "encryptionkey@0"
I20260928 14:54:10.869267 31276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:54:10.870033 31291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:10.870210 31276 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:54:10.870311 31276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "3bbeab0779cd4e518a6c333f5d7cb927"
format_stamp: "Formatted at 2026-09-28 14:54:10 on dist-test-slave-pxl6"
server_key: "056e0e5c9df5d279c29eb8656bbb31b1"
server_key_iv: "3b65fd830a07aa12f90a6d113bd2c267"
server_key_version: "encryptionkey@0"
I20260928 14:54:10.870393 31276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:54:10.888427 31276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:54:10.890627 31276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:54:10.890754 31276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:54:10.890938 31276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:54:10.891278 31276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:54:10.891311 31276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:10.891363 31276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:54:10.891390 31276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:10.897174 31276 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:38899
I20260928 14:54:10.897251 31404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:38899 every 8 connection(s)
I20260928 14:54:10.897562 31276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:54:10.900835 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 31276
I20260928 14:54:10.900918 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:54:10.901132 20946 external_mini_cluster.cc:1428] Setting key 2f442476b7dff853e8b4924f41911b9b
Sep 28 14:54:10 dist-test-slave-pxl6 krb5kdc[31169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607250, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:54:10.907612 31253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:10.898282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:35681 (local address 127.20.116.190:34643)
0928 14:54:10.898357 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:10.898360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:10.898520 (+ 160us) server_negotiation.cc:407] Connection header received
0928 14:54:10.899078 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:10.899079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:10.899090 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:10.899146 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:10.900055 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:10.900578 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:10.901723 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:10.901939 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:10.902034 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:10.904384 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:54:10.904386 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:54:10.904386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:54:10.904395 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:54:10.906170 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:54:10.906636 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:54:10.906642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:54:10.906642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:54:10.906702 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:54:10.906957 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:54:10.906958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:54:10.906958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:54:10.907074 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:54:10.907149 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:10.907431 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 14:54:10.907520 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:54:10.907856 31407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:10.898235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34643 (local address 127.20.116.129:35681)
0928 14:54:10.898401 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:10.898437 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:10.898989 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:10.899203 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:10.899206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:10.899773 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:10.899971 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:10.899975 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:10.900657 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:10.900658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:10.901616 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:10.901618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:10.902045 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:10.902046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:10.902187 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:10.902713 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:10.902721 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:10.904296 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:54:10.906252 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:54:10.906260 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:54:10.906262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:54:10.906488 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:54:10.906773 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:54:10.906774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:54:10.906775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:54:10.906886 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:54:10.907139 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:54:10.907147 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:54:10.907296 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 14:54:10.907395 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 14:54:10.907559 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
I20260928 14:54:10.908334 31405 heartbeater.cc:347] Connected to a master server at 127.20.116.190:34643
I20260928 14:54:10.908425 31405 heartbeater.cc:464] Registering TS with master...
I20260928 14:54:10.908622 31405 heartbeater.cc:511] Master 127.20.116.190:34643 requested a full tablet report, sending...
I20260928 14:54:10.909170 31214 ts_manager.cc:194] Registered new tserver with Master: 3bbeab0779cd4e518a6c333f5d7cb927 (127.20.116.129:38899)
WARNING: no policy specified for kudu/127.20.116.130@KRBTEST.COM; defaulting to no policy
I20260928 14:54:10.909955 31214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:35681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
I20260928 14:54:10.929574 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34643
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:39293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:54:11.013571 31410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:54:11.013798 31410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:54:11.013823 31410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:54:11.015106 31410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:54:11.015156 31410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:54:11.015183 31410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:54:11.016711 31410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:39293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:54:11.016960 31410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:54:11.017213 31410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:54:11.022410 31415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:11.022394 31416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:11.022485 31410 server_base.cc:1034] running on GCE node
W20260928 14:54:11.022394 31418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:11.022915 31410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:54:11.023192 31410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:54:11.024325 31410 hybrid_clock.cc:648] HybridClock initialized: now 1790607251024309 us; error 29 us; skew 500 ppm
Sep 28 14:54:11 dist-test-slave-pxl6 krb5kdc[31169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607251, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:54:11.027160 31410 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:54:11.027630 31410 webserver.cc:460] Webserver started at http://127.20.116.130:39009/ using document root <none> and password file <none>
I20260928 14:54:11.027879 31410 fs_manager.cc:359] Metadata directory not provided
I20260928 14:54:11.027925 31410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:54:11.028074 31410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:54:11.029186 31410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "448c30a67c5f43e3b12d1d4713fee540"
format_stamp: "Formatted at 2026-09-28 14:54:11 on dist-test-slave-pxl6"
server_key: "cf5fb7df5218ae6a014a6c80d7be033a"
server_key_iv: "e990a4648cccc5dfbb1bd1ad761d0154"
server_key_version: "encryptionkey@0"
I20260928 14:54:11.029553 31410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "448c30a67c5f43e3b12d1d4713fee540"
format_stamp: "Formatted at 2026-09-28 14:54:11 on dist-test-slave-pxl6"
server_key: "cf5fb7df5218ae6a014a6c80d7be033a"
server_key_iv: "e990a4648cccc5dfbb1bd1ad761d0154"
server_key_version: "encryptionkey@0"
I20260928 14:54:11.030958 31410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:54:11.031730 31425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:11.031910 31410 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:54:11.032002 31410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "448c30a67c5f43e3b12d1d4713fee540"
format_stamp: "Formatted at 2026-09-28 14:54:11 on dist-test-slave-pxl6"
server_key: "cf5fb7df5218ae6a014a6c80d7be033a"
server_key_iv: "e990a4648cccc5dfbb1bd1ad761d0154"
server_key_version: "encryptionkey@0"
I20260928 14:54:11.032073 31410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:54:11.052043 31410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:54:11.053995 31410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:54:11.054163 31410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:54:11.054418 31410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:54:11.054847 31410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:54:11.054905 31410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:11.054940 31410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:54:11.054971 31410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:11.061514 31410 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:36045
I20260928 14:54:11.061556 31538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:36045 every 8 connection(s)
I20260928 14:54:11.061901 31410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:54:11.064983 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 31410
I20260928 14:54:11.065074 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:54:11.065274 20946 external_mini_cluster.cc:1428] Setting key e5759df5783284402b6046aafd942910
Sep 28 14:54:11 dist-test-slave-pxl6 krb5kdc[31169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607251, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:54:11.071954 31253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:11.062637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:56935 (local address 127.20.116.190:34643)
0928 14:54:11.062710 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:11.062713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:11.062988 (+ 275us) server_negotiation.cc:407] Connection header received
0928 14:54:11.063535 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:11.063536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:11.063548 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:11.063602 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:11.064525 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.065188 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.066453 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.066707 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.066764 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:11.069020 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:54:11.069022 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:54:11.069023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:54:11.069031 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:54:11.070497 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:54:11.070963 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:54:11.070964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:54:11.070964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:54:11.071025 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:54:11.071283 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:54:11.071284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:54:11.071285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:54:11.071405 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:54:11.071479 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:11.071763 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 14:54:11.071859 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 14:54:11.072139 31541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:11.062681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34643 (local address 127.20.116.130:56935)
0928 14:54:11.062855 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:11.062894 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:11.063448 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:11.063658 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:11.063661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:11.064246 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:11.064450 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:11.064455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.065265 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.065266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:11.066380 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:11.066383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.066784 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.066785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:11.066945 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:11.067545 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:11.067556 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:11.068911 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:54:11.070632 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:54:11.070640 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:54:11.070642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:54:11.070885 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:54:11.071097 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:54:11.071098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:54:11.071098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:54:11.071220 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:54:11.071470 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:54:11.071477 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:54:11.071625 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 14:54:11.071729 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 14:54:11.071894 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20260928 14:54:11.072806 31539 heartbeater.cc:347] Connected to a master server at 127.20.116.190:34643
I20260928 14:54:11.072916 31539 heartbeater.cc:464] Registering TS with master...
I20260928 14:54:11.073127 31539 heartbeater.cc:511] Master 127.20.116.190:34643 requested a full tablet report, sending...
I20260928 14:54:11.073547 31214 ts_manager.cc:194] Registered new tserver with Master: 448c30a67c5f43e3b12d1d4713fee540 (127.20.116.130:36045)
WARNING: no policy specified for kudu/127.20.116.131@KRBTEST.COM; defaulting to no policy
I20260928 14:54:11.073961 31214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:56935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
I20260928 14:54:11.093972 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34643
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:39293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:54:11.175781 31544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:54:11.176031 31544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:54:11.176076 31544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:54:11.177273 31544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:54:11.177338 31544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:54:11.177383 31544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:54:11.178893 31544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:39293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:54:11.179168 31544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:54:11.179401 31544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:54:11.184381 31549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:11.184499 31544 server_base.cc:1034] running on GCE node
W20260928 14:54:11.184432 31552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:11.184413 31550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:11.184841 31544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:54:11.185089 31544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:54:11.186241 31544 hybrid_clock.cc:648] HybridClock initialized: now 1790607251186220 us; error 38 us; skew 500 ppm
Sep 28 14:54:11 dist-test-slave-pxl6 krb5kdc[31169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607251, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:54:11.188868 31544 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:54:11.189347 31544 webserver.cc:460] Webserver started at http://127.20.116.131:44559/ using document root <none> and password file <none>
I20260928 14:54:11.189585 31544 fs_manager.cc:359] Metadata directory not provided
I20260928 14:54:11.189648 31544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:54:11.189791 31544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:54:11.190929 31544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "5590051bc31c43fb98b51959b38bd744"
format_stamp: "Formatted at 2026-09-28 14:54:11 on dist-test-slave-pxl6"
server_key: "5fa93f416295e86cc1aa30d1432c7d49"
server_key_iv: "e5d6b4e5ea785ddc4ee79e37a6061a11"
server_key_version: "encryptionkey@0"
I20260928 14:54:11.191315 31544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "5590051bc31c43fb98b51959b38bd744"
format_stamp: "Formatted at 2026-09-28 14:54:11 on dist-test-slave-pxl6"
server_key: "5fa93f416295e86cc1aa30d1432c7d49"
server_key_iv: "e5d6b4e5ea785ddc4ee79e37a6061a11"
server_key_version: "encryptionkey@0"
I20260928 14:54:11.192737 31544 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:54:11.193496 31559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:11.193665 31544 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:54:11.193782 31544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "5590051bc31c43fb98b51959b38bd744"
format_stamp: "Formatted at 2026-09-28 14:54:11 on dist-test-slave-pxl6"
server_key: "5fa93f416295e86cc1aa30d1432c7d49"
server_key_iv: "e5d6b4e5ea785ddc4ee79e37a6061a11"
server_key_version: "encryptionkey@0"
I20260928 14:54:11.193863 31544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:54:11.202474 31544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:54:11.204901 31544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:54:11.205067 31544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:54:11.205340 31544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:54:11.205667 31544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:54:11.205724 31544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:11.205767 31544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:54:11.205794 31544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:11.211917 31544 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:45865
I20260928 14:54:11.211966 31672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:45865 every 8 connection(s)
I20260928 14:54:11.212301 31544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:54:11.218964 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 31544
I20260928 14:54:11.219062 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:54:11.219261 20946 external_mini_cluster.cc:1428] Setting key 7583156b48bfc246eb801afb69065763
Sep 28 14:54:11 dist-test-slave-pxl6 krb5kdc[31169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607251, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:54:11.223011 31253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:11.213044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:46799 (local address 127.20.116.190:34643)
0928 14:54:11.213126 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:11.213128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:11.213331 (+ 203us) server_negotiation.cc:407] Connection header received
0928 14:54:11.213835 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:11.213836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:11.213849 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:11.213907 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:11.214939 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.215698 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.217285 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.217632 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.217730 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:11.220475 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:54:11.220477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:54:11.220477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:54:11.220484 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:54:11.221748 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:54:11.222127 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:54:11.222128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:54:11.222128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:54:11.222183 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:54:11.222404 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:54:11.222405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:54:11.222405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:54:11.222510 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:54:11.222628 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:11.222875 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 14:54:11.222943 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:54:11.223237 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:11.212992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34643 (local address 127.20.116.131:46799)
0928 14:54:11.213177 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:11.213220 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:11.213744 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:11.213963 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:11.213966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:11.214635 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:11.214850 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:11.214855 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.215793 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.215794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:11.217188 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:11.217192 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.217734 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.217735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:11.217896 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:11.218507 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:11.218519 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:11.220399 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:54:11.221825 (+ 1426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:54:11.221831 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:54:11.221833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:54:11.222064 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:54:11.222240 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:54:11.222241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:54:11.222241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:54:11.222350 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:54:11.222617 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:54:11.222622 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:54:11.222763 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 14:54:11.222850 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:54:11.222981 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20260928 14:54:11.223717 31673 heartbeater.cc:347] Connected to a master server at 127.20.116.190:34643
I20260928 14:54:11.223826 31673 heartbeater.cc:464] Registering TS with master...
I20260928 14:54:11.223982 31673 heartbeater.cc:511] Master 127.20.116.190:34643 requested a full tablet report, sending...
I20260928 14:54:11.224351 31214 ts_manager.cc:194] Registered new tserver with Master: 5590051bc31c43fb98b51959b38bd744 (127.20.116.131:45865)
I20260928 14:54:11.224761 31214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:46799
I20260928 14:54:11.231293 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:54:11.239323 31253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:11.232685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45986 (local address 127.20.116.190:34643)
0928 14:54:11.232790 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:11.232792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:11.232856 (+ 64us) server_negotiation.cc:407] Connection header received
0928 14:54:11.232912 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:11.232913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:11.232922 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:11.232980 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:11.233526 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.234033 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.235013 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.235237 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.235297 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:11.236674 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:54:11.236675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:54:11.236676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:54:11.236684 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:54:11.238168 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:54:11.238582 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:54:11.238583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:54:11.238584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:54:11.238648 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:54:11.238816 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:54:11.238817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:54:11.238817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:54:11.238978 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:54:11.239060 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:11.239175 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 14:54:11.239251 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 14:54:11.241039 31214 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:54:11.242774 31214 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:54:11.249609 31684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:11.245404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:45865 (local address 127.0.0.1:35794)
0928 14:54:11.245686 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:11.245733 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:11.245806 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:11.246138 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:11.246142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:11.246144 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:54:11.246251 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:11.246254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.247276 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.247277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:11.248518 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:11.248520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.249270 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.249271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:11.249352 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:11.249366 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 14:54:11.249466 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 14:54:11.249524 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
I20260928 14:54:11.249620 31687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:11.245500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35794 (local address 127.20.116.131:45865)
0928 14:54:11.245783 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:11.245787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:11.245798 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:54:11.245917 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:11.245920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:11.245940 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:11.246028 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:54:11.246341 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.247160 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.248606 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.249182 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.249274 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:11.249339 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:11.249487 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 14:54:11.249538 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":87,"threads_started":1}
I20260928 14:54:11.249967 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:11.245407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:38899 (local address 127.0.0.1:56160)
0928 14:54:11.245852 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:11.245862 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:11.245907 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:11.246191 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:11.246192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:11.246194 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:54:11.246286 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:11.246288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.247689 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.247690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:11.248765 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:11.248767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.249687 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.249688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:11.249763 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:11.249768 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:54:11.249825 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 14:54:11.249868 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":28,"thread_start_us":71,"threads_started":1}
I20260928 14:54:11.250129 31688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:11.245713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:36045 (local address 127.0.0.1:42480)
0928 14:54:11.246063 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:11.246074 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:11.246128 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:11.246397 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:11.246398 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:11.246399 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:54:11.246435 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:11.246437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.247725 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.247726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:11.248688 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:11.248691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.249842 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.249843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:11.249890 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:11.249893 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:54:11.249949 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 14:54:11.249987 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":88,"threads_started":1}
I20260928 14:54:11.250250 31686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:11.245498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56160 (local address 127.20.116.129:38899)
0928 14:54:11.245680 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:11.245686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:11.245963 (+ 277us) server_negotiation.cc:407] Connection header received
0928 14:54:11.246011 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:11.246014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:11.246030 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:11.246101 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:54:11.246371 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.247576 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.248856 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.249550 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.249609 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:11.249653 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:11.250090 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 14:54:11.250135 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":84,"threads_started":1}
I20260928 14:54:11.250463 31607 tablet_service.cc:1467] Processing CreateTablet for tablet 28cd732b5e1446dd903281ce138f6874 (DEFAULT_TABLE table=test-table [id=6c781f1de1cd47ab8fbd9a802412e1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:54:11.250893 31607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28cd732b5e1446dd903281ce138f6874. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:54:11.250991 31339 tablet_service.cc:1467] Processing CreateTablet for tablet 28cd732b5e1446dd903281ce138f6874 (DEFAULT_TABLE table=test-table [id=6c781f1de1cd47ab8fbd9a802412e1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:54:11.251313 31339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28cd732b5e1446dd903281ce138f6874. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:54:11.254721 31690 tablet_bootstrap.cc:492] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744: Bootstrap starting.
I20260928 14:54:11.254788 31689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:11.245868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.20.116.130:36045)
0928 14:54:11.246055 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:11.246060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:11.246184 (+ 124us) server_negotiation.cc:407] Connection header received
0928 14:54:11.246226 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:11.246228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:11.246244 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:11.246313 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:54:11.246516 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.247620 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.248773 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.249506 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.249585 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:11.249641 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:11.254614 (+ 4973us) server_negotiation.cc:299] Negotiation successful
0928 14:54:11.254684 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
I20260928 14:54:11.255074 31691 tablet_bootstrap.cc:492] T 28cd732b5e1446dd903281ce138f6874 P 3bbeab0779cd4e518a6c333f5d7cb927: Bootstrap starting.
I20260928 14:54:11.255489 31473 tablet_service.cc:1467] Processing CreateTablet for tablet 28cd732b5e1446dd903281ce138f6874 (DEFAULT_TABLE table=test-table [id=6c781f1de1cd47ab8fbd9a802412e1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:54:11.255800 31473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28cd732b5e1446dd903281ce138f6874. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:54:11.255822 31690 tablet_bootstrap.cc:654] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744: Neither blocks nor log segments found. Creating new log.
I20260928 14:54:11.255822 31691 tablet_bootstrap.cc:654] T 28cd732b5e1446dd903281ce138f6874 P 3bbeab0779cd4e518a6c333f5d7cb927: Neither blocks nor log segments found. Creating new log.
I20260928 14:54:11.256438 31691 log.cc:824] T 28cd732b5e1446dd903281ce138f6874 P 3bbeab0779cd4e518a6c333f5d7cb927: Log is configured to *not* fsync() on all Append() calls
I20260928 14:54:11.256438 31690 log.cc:824] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744: Log is configured to *not* fsync() on all Append() calls
I20260928 14:54:11.257184 31691 tablet_bootstrap.cc:492] T 28cd732b5e1446dd903281ce138f6874 P 3bbeab0779cd4e518a6c333f5d7cb927: No bootstrap required, opened a new log
I20260928 14:54:11.257243 31691 ts_tablet_manager.cc:1397] T 28cd732b5e1446dd903281ce138f6874 P 3bbeab0779cd4e518a6c333f5d7cb927: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:54:11.257838 31690 tablet_bootstrap.cc:492] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744: No bootstrap required, opened a new log
I20260928 14:54:11.257910 31690 ts_tablet_manager.cc:1397] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:54:11.258829 31694 tablet_bootstrap.cc:492] T 28cd732b5e1446dd903281ce138f6874 P 448c30a67c5f43e3b12d1d4713fee540: Bootstrap starting.
I20260928 14:54:11.259359 31691 raft_consensus.cc:348] T 28cd732b5e1446dd903281ce138f6874 P 3bbeab0779cd4e518a6c333f5d7cb927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bbeab0779cd4e518a6c333f5d7cb927" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38899 } } peers { permanent_uuid: "5590051bc31c43fb98b51959b38bd744" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 45865 } } peers { permanent_uuid: "448c30a67c5f43e3b12d1d4713fee540" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 36045 } }
I20260928 14:54:11.259558 31691 raft_consensus.cc:374] T 28cd732b5e1446dd903281ce138f6874 P 3bbeab0779cd4e518a6c333f5d7cb927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:54:11.259569 31694 tablet_bootstrap.cc:654] T 28cd732b5e1446dd903281ce138f6874 P 448c30a67c5f43e3b12d1d4713fee540: Neither blocks nor log segments found. Creating new log.
I20260928 14:54:11.259595 31691 raft_consensus.cc:729] T 28cd732b5e1446dd903281ce138f6874 P 3bbeab0779cd4e518a6c333f5d7cb927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bbeab0779cd4e518a6c333f5d7cb927, State: Initialized, Role: FOLLOWER
I20260928 14:54:11.259745 31691 consensus_queue.cc:260] T 28cd732b5e1446dd903281ce138f6874 P 3bbeab0779cd4e518a6c333f5d7cb927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bbeab0779cd4e518a6c333f5d7cb927" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38899 } } peers { permanent_uuid: "5590051bc31c43fb98b51959b38bd744" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 45865 } } peers { permanent_uuid: "448c30a67c5f43e3b12d1d4713fee540" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 36045 } }
I20260928 14:54:11.260033 31691 ts_tablet_manager.cc:1428] T 28cd732b5e1446dd903281ce138f6874 P 3bbeab0779cd4e518a6c333f5d7cb927: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 14:54:11.259951 31690 raft_consensus.cc:348] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bbeab0779cd4e518a6c333f5d7cb927" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38899 } } peers { permanent_uuid: "5590051bc31c43fb98b51959b38bd744" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 45865 } } peers { permanent_uuid: "448c30a67c5f43e3b12d1d4713fee540" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 36045 } }
I20260928 14:54:11.260095 31690 raft_consensus.cc:374] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:54:11.260089 31694 log.cc:824] T 28cd732b5e1446dd903281ce138f6874 P 448c30a67c5f43e3b12d1d4713fee540: Log is configured to *not* fsync() on all Append() calls
I20260928 14:54:11.260133 31690 raft_consensus.cc:729] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5590051bc31c43fb98b51959b38bd744, State: Initialized, Role: FOLLOWER
I20260928 14:54:11.260149 31405 heartbeater.cc:503] Master 127.20.116.190:34643 was elected leader, sending a full tablet report...
I20260928 14:54:11.260252 31690 consensus_queue.cc:260] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bbeab0779cd4e518a6c333f5d7cb927" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38899 } } peers { permanent_uuid: "5590051bc31c43fb98b51959b38bd744" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 45865 } } peers { permanent_uuid: "448c30a67c5f43e3b12d1d4713fee540" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 36045 } }
I20260928 14:54:11.260517 31690 ts_tablet_manager.cc:1428] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:54:11.260748 31694 tablet_bootstrap.cc:492] T 28cd732b5e1446dd903281ce138f6874 P 448c30a67c5f43e3b12d1d4713fee540: No bootstrap required, opened a new log
I20260928 14:54:11.260749 31673 heartbeater.cc:503] Master 127.20.116.190:34643 was elected leader, sending a full tablet report...
I20260928 14:54:11.260799 31694 ts_tablet_manager.cc:1397] T 28cd732b5e1446dd903281ce138f6874 P 448c30a67c5f43e3b12d1d4713fee540: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:54:11.262462 31694 raft_consensus.cc:348] T 28cd732b5e1446dd903281ce138f6874 P 448c30a67c5f43e3b12d1d4713fee540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bbeab0779cd4e518a6c333f5d7cb927" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38899 } } peers { permanent_uuid: "5590051bc31c43fb98b51959b38bd744" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 45865 } } peers { permanent_uuid: "448c30a67c5f43e3b12d1d4713fee540" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 36045 } }
I20260928 14:54:11.262665 31694 raft_consensus.cc:374] T 28cd732b5e1446dd903281ce138f6874 P 448c30a67c5f43e3b12d1d4713fee540 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:54:11.262713 31694 raft_consensus.cc:729] T 28cd732b5e1446dd903281ce138f6874 P 448c30a67c5f43e3b12d1d4713fee540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448c30a67c5f43e3b12d1d4713fee540, State: Initialized, Role: FOLLOWER
I20260928 14:54:11.262822 31694 consensus_queue.cc:260] T 28cd732b5e1446dd903281ce138f6874 P 448c30a67c5f43e3b12d1d4713fee540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bbeab0779cd4e518a6c333f5d7cb927" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38899 } } peers { permanent_uuid: "5590051bc31c43fb98b51959b38bd744" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 45865 } } peers { permanent_uuid: "448c30a67c5f43e3b12d1d4713fee540" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 36045 } }
I20260928 14:54:11.263096 31694 ts_tablet_manager.cc:1428] T 28cd732b5e1446dd903281ce138f6874 P 448c30a67c5f43e3b12d1d4713fee540: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:54:11.263127 31539 heartbeater.cc:503] Master 127.20.116.190:34643 was elected leader, sending a full tablet report...
W20260928 14:54:11.312470 31540 tablet.cc:2369] T 28cd732b5e1446dd903281ce138f6874 P 448c30a67c5f43e3b12d1d4713fee540: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:54:11.398272 31406 tablet.cc:2369] T 28cd732b5e1446dd903281ce138f6874 P 3bbeab0779cd4e518a6c333f5d7cb927: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:54:11.415572 31697 raft_consensus.cc:482] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:54:11.415743 31697 raft_consensus.cc:504] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bbeab0779cd4e518a6c333f5d7cb927" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38899 } } peers { permanent_uuid: "5590051bc31c43fb98b51959b38bd744" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 45865 } } peers { permanent_uuid: "448c30a67c5f43e3b12d1d4713fee540" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 36045 } }
I20260928 14:54:11.416230 31697 leader_election.cc:288] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bbeab0779cd4e518a6c333f5d7cb927 (127.20.116.129:38899), 448c30a67c5f43e3b12d1d4713fee540 (127.20.116.130:36045)
I20260928 14:54:11.419310 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:11.416417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:38899 (local address 127.20.116.131:36119)
0928 14:54:11.416480 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:11.416494 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:11.416585 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:11.416875 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:11.416876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:11.416878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:54:11.416956 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:11.416958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.417627 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.417628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:11.418458 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:11.418460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.419051 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.419052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:11.419134 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:11.419139 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:54:11.419194 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 14:54:11.419236 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 14:54:11.419329 31686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:11.416695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:36119 (local address 127.20.116.129:38899)
0928 14:54:11.416746 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:11.416748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:11.416758 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:54:11.416789 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:11.416790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:11.416800 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:11.416857 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:54:11.417039 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.417546 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.418576 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.418981 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.419039 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:11.419046 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:11.419210 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 14:54:11.419247 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 14:54:11.419473 31699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:11.416566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:36045 (local address 127.20.116.131:36747)
0928 14:54:11.416733 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:11.416744 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:11.416805 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:11.416957 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:11.416957 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:11.416959 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:54:11.417043 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:11.417044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.417677 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.417678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:11.418480 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:11.418482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.419215 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.419216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:11.419283 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:11.419287 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:54:11.419345 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 14:54:11.419383 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":45,"thread_start_us":104,"threads_started":1}
I20260928 14:54:11.419480 31689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:11.416681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:36747 (local address 127.20.116.130:36045)
0928 14:54:11.416729 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:11.416731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:11.416805 (+ 74us) server_negotiation.cc:407] Connection header received
0928 14:54:11.416874 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:11.416875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:11.416884 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:11.416952 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:54:11.417117 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.417595 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.418624 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.419129 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.419214 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:11.419222 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:11.419359 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 14:54:11.419397 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 14:54:11.419670 31359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28cd732b5e1446dd903281ce138f6874" candidate_uuid: "5590051bc31c43fb98b51959b38bd744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bbeab0779cd4e518a6c333f5d7cb927" is_pre_election: true
I20260928 14:54:11.419812 31359 raft_consensus.cc:2454] T 28cd732b5e1446dd903281ce138f6874 P 3bbeab0779cd4e518a6c333f5d7cb927 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5590051bc31c43fb98b51959b38bd744 in term 0.
I20260928 14:54:11.420011 31562 leader_election.cc:302] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bbeab0779cd4e518a6c333f5d7cb927, 5590051bc31c43fb98b51959b38bd744; no voters:
I20260928 14:54:11.420146 31697 raft_consensus.cc:2781] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:54:11.420217 31697 raft_consensus.cc:482] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:54:11.420243 31697 raft_consensus.cc:3037] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:54:11.420382 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28cd732b5e1446dd903281ce138f6874" candidate_uuid: "5590051bc31c43fb98b51959b38bd744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448c30a67c5f43e3b12d1d4713fee540" is_pre_election: true
I20260928 14:54:11.420504 31493 raft_consensus.cc:2454] T 28cd732b5e1446dd903281ce138f6874 P 448c30a67c5f43e3b12d1d4713fee540 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5590051bc31c43fb98b51959b38bd744 in term 0.
I20260928 14:54:11.421018 31697 raft_consensus.cc:504] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bbeab0779cd4e518a6c333f5d7cb927" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38899 } } peers { permanent_uuid: "5590051bc31c43fb98b51959b38bd744" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 45865 } } peers { permanent_uuid: "448c30a67c5f43e3b12d1d4713fee540" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 36045 } }
I20260928 14:54:11.421149 31697 leader_election.cc:288] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744 [CANDIDATE]: Term 1 election: Requested vote from peers 3bbeab0779cd4e518a6c333f5d7cb927 (127.20.116.129:38899), 448c30a67c5f43e3b12d1d4713fee540 (127.20.116.130:36045)
I20260928 14:54:11.421367 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28cd732b5e1446dd903281ce138f6874" candidate_uuid: "5590051bc31c43fb98b51959b38bd744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448c30a67c5f43e3b12d1d4713fee540"
I20260928 14:54:11.421370 31359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28cd732b5e1446dd903281ce138f6874" candidate_uuid: "5590051bc31c43fb98b51959b38bd744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bbeab0779cd4e518a6c333f5d7cb927"
I20260928 14:54:11.421455 31493 raft_consensus.cc:3037] T 28cd732b5e1446dd903281ce138f6874 P 448c30a67c5f43e3b12d1d4713fee540 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:54:11.421455 31359 raft_consensus.cc:3037] T 28cd732b5e1446dd903281ce138f6874 P 3bbeab0779cd4e518a6c333f5d7cb927 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:54:11.422417 31359 raft_consensus.cc:2454] T 28cd732b5e1446dd903281ce138f6874 P 3bbeab0779cd4e518a6c333f5d7cb927 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5590051bc31c43fb98b51959b38bd744 in term 1.
I20260928 14:54:11.422417 31493 raft_consensus.cc:2454] T 28cd732b5e1446dd903281ce138f6874 P 448c30a67c5f43e3b12d1d4713fee540 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5590051bc31c43fb98b51959b38bd744 in term 1.
I20260928 14:54:11.422601 31562 leader_election.cc:302] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 448c30a67c5f43e3b12d1d4713fee540, 5590051bc31c43fb98b51959b38bd744; no voters:
I20260928 14:54:11.422746 31697 raft_consensus.cc:2781] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:54:11.422935 31697 raft_consensus.cc:686] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744 [term 1 LEADER]: Becoming Leader. State: Replica: 5590051bc31c43fb98b51959b38bd744, State: Running, Role: LEADER
I20260928 14:54:11.423043 31697 consensus_queue.cc:237] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bbeab0779cd4e518a6c333f5d7cb927" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38899 } } peers { permanent_uuid: "5590051bc31c43fb98b51959b38bd744" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 45865 } } peers { permanent_uuid: "448c30a67c5f43e3b12d1d4713fee540" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 36045 } }
I20260928 14:54:11.424010 31213 catalog_manager.cc:5495] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5590051bc31c43fb98b51959b38bd744 (127.20.116.131). New cstate: current_term: 1 leader_uuid: "5590051bc31c43fb98b51959b38bd744" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bbeab0779cd4e518a6c333f5d7cb927" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5590051bc31c43fb98b51959b38bd744" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "448c30a67c5f43e3b12d1d4713fee540" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 36045 } health_report { overall_health: UNKNOWN } } }
I20260928 14:54:11.436609 31687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:11.433645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35800 (local address 127.20.116.131:45865)
0928 14:54:11.433737 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:11.433740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:11.433749 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:54:11.433784 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:11.433785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:11.433804 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:11.433882 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:54:11.434290 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.435095 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.435913 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:11.436105 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:11.436168 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:11.436299 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:54:11.436403 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:54:11.436467 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:11.436498 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 14:54:11.436517 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 14:54:11.438823 31359 raft_consensus.cc:1261] T 28cd732b5e1446dd903281ce138f6874 P 3bbeab0779cd4e518a6c333f5d7cb927 [term 1 FOLLOWER]: Refusing update from remote peer 5590051bc31c43fb98b51959b38bd744: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:54:11.438895 31493 raft_consensus.cc:1261] T 28cd732b5e1446dd903281ce138f6874 P 448c30a67c5f43e3b12d1d4713fee540 [term 1 FOLLOWER]: Refusing update from remote peer 5590051bc31c43fb98b51959b38bd744: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:54:11.439244 31697 consensus_queue.cc:1035] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "448c30a67c5f43e3b12d1d4713fee540" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 36045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:54:11.439422 31700 consensus_queue.cc:1035] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bbeab0779cd4e518a6c333f5d7cb927" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:54:11.441838 31705 mvcc.cc:205] Tried to move back new op lower bound from 7334327301889445888 to 7334327301829586944. Current Snapshot: MvccSnapshot[applied={T|T < 7334327301889445888}]
I20260928 14:54:11.442973 31704 mvcc.cc:205] Tried to move back new op lower bound from 7334327301889445888 to 7334327301829586944. Current Snapshot: MvccSnapshot[applied={T|T < 7334327301889445888 or (T in {7334327301889445888})}]
I20260928 14:54:11.444082 31213 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:54:11.444186 31213 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:54:11.445109 31213 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46830767e2ed4029af9f0aa529a5b1e2: Sending DeleteTablet for 3 replicas of tablet 28cd732b5e1446dd903281ce138f6874
I20260928 14:54:11.445467 31607 tablet_service.cc:1514] Processing DeleteTablet for tablet 28cd732b5e1446dd903281ce138f6874 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:54:11 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:35794
I20260928 14:54:11.445488 31473 tablet_service.cc:1514] Processing DeleteTablet for tablet 28cd732b5e1446dd903281ce138f6874 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:54:11 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:42480
I20260928 14:54:11.445782 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 31276
I20260928 14:54:11.445796 31339 tablet_service.cc:1514] Processing DeleteTablet for tablet 28cd732b5e1446dd903281ce138f6874 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:54:11 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:56160
I20260928 14:54:11.446991 31714 tablet_replica.cc:330] T 28cd732b5e1446dd903281ce138f6874 P 448c30a67c5f43e3b12d1d4713fee540: stopping tablet replica
I20260928 14:54:11.447111 31714 raft_consensus.cc:2229] T 28cd732b5e1446dd903281ce138f6874 P 448c30a67c5f43e3b12d1d4713fee540 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:54:11.447221 31714 raft_consensus.cc:2258] T 28cd732b5e1446dd903281ce138f6874 P 448c30a67c5f43e3b12d1d4713fee540 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:54:11.447374 31713 tablet_replica.cc:330] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744: stopping tablet replica
I20260928 14:54:11.447464 31713 raft_consensus.cc:2229] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:54:11.447556 31713 raft_consensus.cc:2258] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:54:11.447688 31714 ts_tablet_manager.cc:1905] T 28cd732b5e1446dd903281ce138f6874 P 448c30a67c5f43e3b12d1d4713fee540: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:54:11.447973 31713 ts_tablet_manager.cc:1905] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:54:11.449443 31713 ts_tablet_manager.cc:1918] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:54:11.449446 31714 ts_tablet_manager.cc:1918] T 28cd732b5e1446dd903281ce138f6874 P 448c30a67c5f43e3b12d1d4713fee540: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:54:11.449517 31713 log.cc:1196] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/28cd732b5e1446dd903281ce138f6874
I20260928 14:54:11.449532 31714 log.cc:1196] T 28cd732b5e1446dd903281ce138f6874 P 448c30a67c5f43e3b12d1d4713fee540: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/28cd732b5e1446dd903281ce138f6874
I20260928 14:54:11.449785 31713 ts_tablet_manager.cc:1939] T 28cd732b5e1446dd903281ce138f6874 P 5590051bc31c43fb98b51959b38bd744: Deleting consensus metadata
I20260928 14:54:11.449889 31714 ts_tablet_manager.cc:1939] T 28cd732b5e1446dd903281ce138f6874 P 448c30a67c5f43e3b12d1d4713fee540: Deleting consensus metadata
I20260928 14:54:11.450321 31200 catalog_manager.cc:4841] TS 448c30a67c5f43e3b12d1d4713fee540 (127.20.116.130:36045): tablet 28cd732b5e1446dd903281ce138f6874 (table test-table [id=6c781f1de1cd47ab8fbd9a802412e1ac]) successfully deleted
I20260928 14:54:11.450351 31198 catalog_manager.cc:4841] TS 5590051bc31c43fb98b51959b38bd744 (127.20.116.131:45865): tablet 28cd732b5e1446dd903281ce138f6874 (table test-table [id=6c781f1de1cd47ab8fbd9a802412e1ac]) successfully deleted
W20260928 14:54:11.452165 31200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.116.129:38899 (error 108)
I20260928 14:54:11.452682 31688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:11.452541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:38899 (local address 127.0.0.1:56172)
0928 14:54:11.452595 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:11.452644 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.129:38899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 14:54:11.452802 31200 catalog_manager.cc:4584] TS 3bbeab0779cd4e518a6c333f5d7cb927 (127.20.116.129:38899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 28cd732b5e1446dd903281ce138f6874: Network error: Client connection negotiation failed: client connection to 127.20.116.129:38899: connect: Connection refused (error 111)
I20260928 14:54:11.453002 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 31410
I20260928 14:54:11.458271 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 31544
I20260928 14:54:11.462880 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 31182
2026-09-28T14:54:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2073 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:54:11 dist-test-slave-pxl6 krb5kdc[31718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:54:11 dist-test-slave-pxl6 krb5kdc[31718](info): set up 2 sockets
Sep 28 14:54:11 dist-test-slave-pxl6 krb5kdc[31718](info): commencing operation
krb5kdc: starting...
W20260928 14:54:12.509080 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:54:12 dist-test-slave-pxl6 krb5kdc[31718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:54:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:54:12Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/rangeradmin_127.20.116.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/rangeradmin_127.20.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/rangerlookup_127.20.116.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/rangerlookup_127.20.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/HTTP_127.20.116.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/HTTP_127.20.116.148@KRBTEST.COM.keytab.
I20260928 14:54:12.686866 20946 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHOCNdC/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/postgres -l logfile start
2026-09-28 14:54:15.421 UTC [31748] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 14:54:15.421 UTC [31748] LOG: listening on IPv4 address "127.20.116.148", port 55879
2026-09-28 14:54:15.425 UTC [31748] LOG: listening on Unix socket "/tmp/.s.PGSQL.55879"
2026-09-28 14:54:15.461 UTC [31751] LOG: database system was shut down at 2026-09-28 14:54:13 UTC
2026-09-28 14:54:15.465 UTC [31748] LOG: database system is ready to accept connections
I20260928 14:54:16.420615 20946 mini_postgres.cc:96] Postgres bound to 55879
2026-09-28 14:54:16.427 UTC [31759] FATAL: database "slave" does not exist
127.20.116.148:55879 - accepting connections
I20260928 14:54:16.427744 20946 mini_ranger.cc:161] Starting Ranger...
I20260928 14:54:16.443324 20946 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 14:54:16.978643 20946 mini_ranger.cc:87] Created ranger Postgres database
I20260928 14:54:16.978734 20946 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ranger-admin
2026-09-28 14:54:17,951 [I] DB FLAVOR :POSTGRES
2026-09-28 14:54:17,951 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:54:17,951 [I] Checking connection..
2026-09-28 14:54:17,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 14:54:18,382 [I] Checking connection passed.
2026-09-28 14:54:18,382 [I] --------- Verifying version history table ---------
2026-09-28 14:54:18,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:54:18,635 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:54:18,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:54:18,898 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:54:18,898 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 14:54:18,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 14:54:19.165 UTC [31823] WARNING: there is no transaction in progress
2026-09-28 14:54:19,171 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 14:54:19,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:54:19,407 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 14:54:19,408 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:54:19,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 14:54:19,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 14:54:19,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pxl6.c.gcp-upstream.internal','N') ;"
2026-09-28 14:54:20,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:54:20,419 [I] Table x_portal_user does not exist in database ranger
2026-09-28 14:54:20,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:54:20,684 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 14:54:20,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:54:20,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:54:21,185 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 14:54:21,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 14:54:22.236 UTC [31931] WARNING: there is no transaction in progress
2026-09-28 14:54:22.253 UTC [31931] WARNING: there is no transaction in progress
2026-09-28 14:54:22.278 UTC [31931] WARNING: there is no transaction in progress
2026-09-28 14:54:22.295 UTC [31931] WARNING: there is no transaction in progress
2026-09-28 14:54:22.312 UTC [31931] WARNING: there is no transaction in progress
2026-09-28 14:54:22.331 UTC [31931] WARNING: there is no transaction in progress
2026-09-28 14:54:22.395 UTC [31931] WARNING: there is no transaction in progress
2026-09-28 14:54:22.402 UTC [31931] WARNING: there is no transaction in progress
2026-09-28 14:54:22.418 UTC [31931] WARNING: there is no transaction in progress
2026-09-28 14:54:22.428 UTC [31931] WARNING: there is no transaction in progress
2026-09-28 14:54:22.439 UTC [31931] WARNING: there is no transaction in progress
2026-09-28 14:54:22.455 UTC [31931] WARNING: there is no transaction in progress
2026-09-28 14:54:22.465 UTC [31931] WARNING: there is no transaction in progress
2026-09-28 14:54:22.472 UTC [31931] WARNING: there is no transaction in progress
2026-09-28 14:54:22.479 UTC [31931] WARNING: there is no transaction in progress
2026-09-28 14:54:22.487 UTC [31931] WARNING: there is no transaction in progress
2026-09-28 14:54:23,187 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 14:54:23,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 14:54:23,444 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 14:54:23,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:54:23,711 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 14:54:23,712 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:54:23,980 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 14:54:23,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:54:24,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:54:24,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pxl6.c.gcp-upstream.internal';"
2026-09-28 14:54:24,718 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 14:54:24,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 14:54:24,989 [I] DB_PATCHES have already been applied
I20260928 14:54:24.995457 20946 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHOCNdC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 14:54:24.995533 20946 mini_ranger.cc:193] Using host: 127.20.116.148
I20260928 14:54:24.998893 20946 mini_ranger.cc:227] Ranger admin URL: http://127.20.116.148:48535
Sep 28, 2026 2:54:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:54:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:54:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:54:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:54:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:54:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:54:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 14:54:27.232285 20946 mini_ranger.cc:160] Time spent starting Ranger: real 10.805s user 0.002s sys 0.009s
Sep 28, 2026 2:54:46 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:54:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:54:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:54:47 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:54:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:54:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:54:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:54:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:54:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:54:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:54:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:54:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:54:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:54:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:54:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:54:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:54:47 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:54:48 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 14:54:48.693313 20946 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/rangerkms_127.20.116.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/rangerkms_127.20.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.116.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.116.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.116.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/HTTP_127.20.116.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.116.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/HTTP_127.20.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 14:54:48 dist-test-slave-pxl6 krb5kdc[31718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607288, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 14:54:48.780277 20946 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 14:54:48.790020 20946 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 14:54:49.770768 20946 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 14:54:49.770851 20946 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ranger-kms
I20260928 14:54:49.770874 20946 mini_ranger_kms.cc:227] Using postgres at 127.20.116.148:55879
2026-09-28 14:54:50,190 [I] DB FLAVOR :POSTGRES
2026-09-28 14:54:50,190 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:54:50,191 [I] Checking connection
2026-09-28 14:54:50,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 14:54:50,428 [I] connection success
2026-09-28 14:54:50,428 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 14:54:50,429 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 14:54:50,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 14:54:50,665 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 14:54:50,665 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:54:50,665 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 14:54:50,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55879/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 14:54:50,930 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 14:54:51.460235 20946 mini_ranger_kms.cc:310] Created kms service
I20260928 14:54:51.694453 20946 mini_ranger_kms.cc:326] Created kudu user
I20260928 14:54:51.752964 20946 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 14:54:51 dist-test-slave-pxl6 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607288, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.116.148@KRBTEST.COM
I20260928 14:54:52.239372 20946 mini_ranger_kms.cc:384] Added ranger policy
I20260928 14:54:52.239641 20946 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 14:54:52.239672 20946 mini_ranger_kms.cc:240] Using host: 127.20.116.148
I20260928 14:54:52.243325 20946 mini_ranger_kms.cc:276] Ranger KMS PID: 32113
I20260928 14:54:52.243458 20946 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.116.148:39267
Sep 28, 2026 2:54:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:54:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:54:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:54:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:54:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:54:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 14:54:54.480644 20946 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.700s user 0.008s sys 0.180s
I20260928 14:54:54.480762 20946 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 14:54:54.480842 20946 mini_ranger_kms.cc:401] 127.20.116.148:39267/kms/v1/keys
Sep 28 14:54:54 dist-test-slave-pxl6 krb5kdc[31718](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607294, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.116.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 14:54:54 dist-test-slave-pxl6 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607294, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.116.148@KRBTEST.COM for HTTP/127.20.116.148@KRBTEST.COM
Sep 28 14:54:55 dist-test-slave-pxl6 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607294, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.116.148@KRBTEST.COM for HTTP/127.20.116.148@KRBTEST.COM
Sep 28 14:54:57 dist-test-slave-pxl6 krb5kdc[31718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:54:57.673985 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:35569
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:34039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:35569
--ranger_config_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.116.148:39267/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:54:57.761756 32163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:54:57.761986 32163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:54:57.762010 32163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:54:57.763341 32163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:54:57.763388 32163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:54:57.763403 32163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:54:57.763417 32163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:54:57.763464 32163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:54:57.765121 32163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:34039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.116.148:39267/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:35569
--ranger_config_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:35569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:54:57.765422 32163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:54:57.765690 32163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:54:57.771773 32171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:57.771770 32168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:57.771773 32169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:57.772094 32163 server_base.cc:1034] running on GCE node
I20260928 14:54:57.772452 32163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:54:57.772850 32163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:54:57.774019 32163 hybrid_clock.cc:648] HybridClock initialized: now 1790607297773995 us; error 41 us; skew 500 ppm
Sep 28 14:54:57 dist-test-slave-pxl6 krb5kdc[31718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607297, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:54:57.776952 32163 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:54:57.777565 32163 webserver.cc:460] Webserver started at http://127.20.116.190:41825/ using document root <none> and password file <none>
I20260928 14:54:57.777839 32163 fs_manager.cc:359] Metadata directory not provided
I20260928 14:54:57.777905 32163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:54:57.778044 32163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:54:57 dist-test-slave-pxl6 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607297, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for HTTP/127.20.116.148@KRBTEST.COM
I20260928 14:54:57.825126 32163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "54bdb44f4b1f4be8bacdb7c49fdd650b"
format_stamp: "Formatted at 2026-09-28 14:54:57 on dist-test-slave-pxl6"
server_key: "0aa71545ef86ebf53d7c92b9bc00c025"
server_key_iv: "11aa8e87626ebc94f99c25a5856c42b4"
server_key_version: "kuduclusterkey@0"
I20260928 14:54:57.825562 32163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "54bdb44f4b1f4be8bacdb7c49fdd650b"
format_stamp: "Formatted at 2026-09-28 14:54:57 on dist-test-slave-pxl6"
server_key: "0aa71545ef86ebf53d7c92b9bc00c025"
server_key_iv: "11aa8e87626ebc94f99c25a5856c42b4"
server_key_version: "kuduclusterkey@0"
I20260928 14:54:57.827106 32163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:54:57.843109 32178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:57.843667 32163 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:54:57.843793 32163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "54bdb44f4b1f4be8bacdb7c49fdd650b"
format_stamp: "Formatted at 2026-09-28 14:54:57 on dist-test-slave-pxl6"
server_key: "0aa71545ef86ebf53d7c92b9bc00c025"
server_key_iv: "11aa8e87626ebc94f99c25a5856c42b4"
server_key_version: "kuduclusterkey@0"
I20260928 14:54:57.843914 32163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:54:57.857113 32163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:54:57.859144 32163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:54:57.859323 32163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:54:57.864464 32163 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:35569
I20260928 14:54:57.864483 32230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:35569 every 8 connection(s)
I20260928 14:54:57.864949 32163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:54:57.865665 32231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:54:57.868101 32231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b: Bootstrap starting.
I20260928 14:54:57.868711 32231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b: Neither blocks nor log segments found. Creating new log.
I20260928 14:54:57.869010 32231 log.cc:824] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b: Log is configured to *not* fsync() on all Append() calls
I20260928 14:54:57.869705 32231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b: No bootstrap required, opened a new log
I20260928 14:54:57.870034 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 32163
I20260928 14:54:57.870142 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:54:57.870337 20946 external_mini_cluster.cc:1428] Setting key 208d3f6fc5acc1df1756b893962aea0f
I20260928 14:54:57.872236 32231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54bdb44f4b1f4be8bacdb7c49fdd650b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35569 } }
I20260928 14:54:57.872406 32231 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:54:57.872469 32231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54bdb44f4b1f4be8bacdb7c49fdd650b, State: Initialized, Role: FOLLOWER
I20260928 14:54:57.872637 32231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54bdb44f4b1f4be8bacdb7c49fdd650b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35569 } }
I20260928 14:54:57.872737 32231 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:54:57.872771 32231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:54:57.872845 32231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:54:57.873605 32231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54bdb44f4b1f4be8bacdb7c49fdd650b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35569 } }
I20260928 14:54:57.873749 32231 leader_election.cc:302] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54bdb44f4b1f4be8bacdb7c49fdd650b; no voters:
I20260928 14:54:57.873961 32231 leader_election.cc:288] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:54:57.873989 32236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:54:57.874218 32236 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b [term 1 LEADER]: Becoming Leader. State: Replica: 54bdb44f4b1f4be8bacdb7c49fdd650b, State: Running, Role: LEADER
I20260928 14:54:57.874362 32231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:54:57.874346 32236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54bdb44f4b1f4be8bacdb7c49fdd650b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35569 } }
I20260928 14:54:57.874800 32236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54bdb44f4b1f4be8bacdb7c49fdd650b. Latest consensus state: current_term: 1 leader_uuid: "54bdb44f4b1f4be8bacdb7c49fdd650b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54bdb44f4b1f4be8bacdb7c49fdd650b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35569 } } }
I20260928 14:54:57.874964 32236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b [sys.catalog]: This master's current role is: LEADER
I20260928 14:54:57.875201 32238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54bdb44f4b1f4be8bacdb7c49fdd650b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54bdb44f4b1f4be8bacdb7c49fdd650b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35569 } } }
I20260928 14:54:57.875313 32238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b [sys.catalog]: This master's current role is: LEADER
I20260928 14:54:57.875509 32245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:54:57.876120 32245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:54:57.876458 32231 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 28 14:54:57 dist-test-slave-pxl6 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:54:57.879835 32253 mvcc.cc:205] Tried to move back new op lower bound from 7334327492104790016 to 7334327492094189568. Current Snapshot: MvccSnapshot[applied={T|T < 7334327492104790016}]
I20260928 14:54:57.880442 32245 catalog_manager.cc:1347] Generated new cluster ID: c70efa83c41041eeb45f101821498975
I20260928 14:54:57.880496 32245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:54:57.883762 32235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:57.870908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.20.116.190:35569)
0928 14:54:57.871104 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:57.871113 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:57.871147 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:54:57.872246 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:57.872258 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:57.872360 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:57.872520 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:57.873407 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:57.874911 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:57.875990 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:57.876638 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:57.878608 (+ 1970us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:57.879787 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:54:57.879791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:54:57.879791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:54:57.879802 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:54:57.881774 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:54:57.882185 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:54:57.882189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:54:57.882189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:54:57.882302 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:54:57.882496 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:54:57.882497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:54:57.882497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:54:57.882835 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:54:57.882997 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:57.883437 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 14:54:57.883553 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
I20260928 14:54:57.908660 32245 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:54:57.909598 32245 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:54:57.912006 32245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b: Generated new TSK 0
I20260928 14:54:57.912290 32245 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
I20260928 14:54:58.749801 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.116.148:39267/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35569
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:34039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:54:58.862926 32276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:54:58.863293 32276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:54:58.863371 32276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:54:58.867821 32276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:54:58.867955 32276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:54:58.868042 32276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:54:58.870813 32276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:34039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.116.148:39267/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:54:58.871374 32276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:54:58.871770 32276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:54:58.879195 32284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:58.879197 32281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:58.879317 32282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:58.881003 32276 server_base.cc:1034] running on GCE node
I20260928 14:54:58.881366 32276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:54:58.881654 32276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:54:58.882818 32276 hybrid_clock.cc:648] HybridClock initialized: now 1790607298882782 us; error 48 us; skew 500 ppm
Sep 28 14:54:58 dist-test-slave-pxl6 krb5kdc[31718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607298, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:54:58.886205 32276 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:54:58.886802 32276 webserver.cc:460] Webserver started at http://127.20.116.129:33159/ using document root <none> and password file <none>
I20260928 14:54:58.887120 32276 fs_manager.cc:359] Metadata directory not provided
I20260928 14:54:58.887188 32276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:54:58.887352 32276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:54:58 dist-test-slave-pxl6 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607298, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for HTTP/127.20.116.148@KRBTEST.COM
I20260928 14:54:58.901803 32276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "4b607efee321476896c686337df987f4"
format_stamp: "Formatted at 2026-09-28 14:54:58 on dist-test-slave-pxl6"
server_key: "63627519f0f5464aa13ff5e8c9d14404"
server_key_iv: "d2c245f929b44564589a76649670dfa7"
server_key_version: "kuduclusterkey@0"
I20260928 14:54:58.902285 32276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "4b607efee321476896c686337df987f4"
format_stamp: "Formatted at 2026-09-28 14:54:58 on dist-test-slave-pxl6"
server_key: "63627519f0f5464aa13ff5e8c9d14404"
server_key_iv: "d2c245f929b44564589a76649670dfa7"
server_key_version: "kuduclusterkey@0"
I20260928 14:54:58.904163 32276 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 14:54:58.919981 32291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:58.920183 32276 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:54:58.920281 32276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "4b607efee321476896c686337df987f4"
format_stamp: "Formatted at 2026-09-28 14:54:58 on dist-test-slave-pxl6"
server_key: "63627519f0f5464aa13ff5e8c9d14404"
server_key_iv: "d2c245f929b44564589a76649670dfa7"
server_key_version: "kuduclusterkey@0"
I20260928 14:54:58.920388 32276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:54:58.936983 32276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:54:58.939662 32276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:54:58.939864 32276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:54:58.940137 32276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:54:58.940552 32276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:54:58.940639 32276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:58.940754 32276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:54:58.940831 32276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:58.948216 32276 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:37577
I20260928 14:54:58.948781 32276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:54:58.949419 32404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:37577 every 8 connection(s)
I20260928 14:54:58.952436 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 32276
I20260928 14:54:58.952548 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:54:58.952755 20946 external_mini_cluster.cc:1428] Setting key 49485f33dadf6c608b15dfc2e3fb6e2e
Sep 28 14:54:58 dist-test-slave-pxl6 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607298, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:54:58.963469 32407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:58.950196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35569 (local address 127.20.116.129:45195)
0928 14:54:58.950769 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:58.950814 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:58.951509 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:58.951974 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:58.951979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:58.952226 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:58.952538 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:58.952545 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:58.953629 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:58.953630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:58.955082 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:58.955086 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:58.956514 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:58.956514 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:58.956664 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:58.957286 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:58.957298 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:58.959165 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:54:58.961538 (+ 2373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:54:58.961548 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:54:58.961551 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:54:58.961915 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:54:58.962168 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:54:58.962169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:54:58.962169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:54:58.962325 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:54:58.962693 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:54:58.962701 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:54:58.962908 (+ 207us) client_negotiation.cc:769] Sending connection context
0928 14:54:58.963019 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 14:54:58.963197 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":87,"threads_started":1}
I20260928 14:54:58.964241 32408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:58.950495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:45195 (local address 127.20.116.190:35569)
0928 14:54:58.951702 (+ 1207us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:58.951704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:58.951726 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:54:58.951756 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:58.951757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:58.951772 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:58.951834 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:58.952660 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:58.953524 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:58.956013 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:58.956341 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:58.956395 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:58.959302 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:54:58.959305 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:54:58.959305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:54:58.959315 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:54:58.961421 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:54:58.962021 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:54:58.962023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:54:58.962023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:54:58.962097 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:54:58.962397 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:54:58.962398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:54:58.962399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:54:58.962599 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:54:58.962663 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:58.964020 (+ 1357us) server_negotiation.cc:299] Negotiation successful
0928 14:54:58.964134 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1182,"thread_start_us":85,"threads_started":1}
I20260928 14:54:58.965085 32405 heartbeater.cc:347] Connected to a master server at 127.20.116.190:35569
I20260928 14:54:58.965189 32405 heartbeater.cc:464] Registering TS with master...
I20260928 14:54:58.965402 32405 heartbeater.cc:511] Master 127.20.116.190:35569 requested a full tablet report, sending...
I20260928 14:54:58.965961 32195 ts_manager.cc:194] Registered new tserver with Master: 4b607efee321476896c686337df987f4 (127.20.116.129:37577)
I20260928 14:54:58.966867 32195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:45195
WARNING: no policy specified for kudu/127.20.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
I20260928 14:54:59.002859 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.116.148:39267/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35569
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:34039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:54:59.128113 32413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:54:59.128494 32413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:54:59.128572 32413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:54:59.130370 32413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:54:59.130692 32413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:54:59.130798 32413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:54:59.133342 32413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:34039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.116.148:39267/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:54:59.133844 32413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:54:59.134169 32413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:54:59.140800 32423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:59.142653 32420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:59.141863 32413 server_base.cc:1034] running on GCE node
W20260928 14:54:59.143153 32421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:59.143427 32413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:54:59.143698 32413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:54:59.144867 32413 hybrid_clock.cc:648] HybridClock initialized: now 1790607299144832 us; error 47 us; skew 500 ppm
Sep 28 14:54:59 dist-test-slave-pxl6 krb5kdc[31718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607299, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:54:59.148181 32413 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:54:59.148717 32413 webserver.cc:460] Webserver started at http://127.20.116.130:37641/ using document root <none> and password file <none>
I20260928 14:54:59.149025 32413 fs_manager.cc:359] Metadata directory not provided
I20260928 14:54:59.149152 32413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:54:59.149335 32413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:54:59 dist-test-slave-pxl6 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607299, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for HTTP/127.20.116.148@KRBTEST.COM
I20260928 14:54:59.166608 32413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "918cd143ff1c42aeac93a4064b5272af"
format_stamp: "Formatted at 2026-09-28 14:54:59 on dist-test-slave-pxl6"
server_key: "1c88804b215f0b19d6a512bb16d2b62b"
server_key_iv: "21e132ec529cccb4b5c0dea7024b966a"
server_key_version: "kuduclusterkey@0"
I20260928 14:54:59.167225 32413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "918cd143ff1c42aeac93a4064b5272af"
format_stamp: "Formatted at 2026-09-28 14:54:59 on dist-test-slave-pxl6"
server_key: "1c88804b215f0b19d6a512bb16d2b62b"
server_key_iv: "21e132ec529cccb4b5c0dea7024b966a"
server_key_version: "kuduclusterkey@0"
I20260928 14:54:59.169230 32413 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
Sep 28 14:54:59 dist-test-slave-pxl6 krb5kdc[31718](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607299, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:54:59.190615 32430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:59.190908 32413 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260928 14:54:59.191020 32413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "918cd143ff1c42aeac93a4064b5272af"
format_stamp: "Formatted at 2026-09-28 14:54:59 on dist-test-slave-pxl6"
server_key: "1c88804b215f0b19d6a512bb16d2b62b"
server_key_iv: "21e132ec529cccb4b5c0dea7024b966a"
server_key_version: "kuduclusterkey@0"
I20260928 14:54:59.191138 32413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:54:59.230376 32413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:54:59.233959 32413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:54:59.234198 32413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:54:59.234490 32413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:54:59.234923 32413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:54:59.234999 32413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:59.235066 32413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:54:59.235136 32413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:59.241503 32413 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:40825
I20260928 14:54:59.242036 32413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:54:59.242156 32543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:40825 every 8 connection(s)
Sep 28 14:54:59 dist-test-slave-pxl6 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607299, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:54:59.250828 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 32413
I20260928 14:54:59.250957 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:54:59.251184 20946 external_mini_cluster.cc:1428] Setting key 36a2aa610b752133fc8f38913cf89c01
I20260928 14:54:59.254774 32408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:59.243128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:44483 (local address 127.20.116.190:35569)
0928 14:54:59.243182 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:59.243185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:59.243447 (+ 262us) server_negotiation.cc:407] Connection header received
0928 14:54:59.244190 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:59.244192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:59.244208 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:59.244258 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:59.245094 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.245896 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.247541 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.247914 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.248006 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:59.250830 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:54:59.250832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:54:59.250832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:54:59.250842 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:54:59.252831 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:54:59.253425 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:54:59.253427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:54:59.253427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:54:59.253496 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:54:59.253853 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:54:59.253854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:54:59.253854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:54:59.254007 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:54:59.254053 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:59.254496 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 14:54:59.254654 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 14:54:59.255393 32546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:59.243077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35569 (local address 127.20.116.130:44483)
0928 14:54:59.243270 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:59.243326 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:59.244045 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:59.244362 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:59.244367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:59.244663 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:59.244984 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:59.244992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.246014 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.246016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:59.247438 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:59.247442 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.248022 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.248023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:59.248192 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:59.248861 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:59.248872 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:59.250714 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:54:59.252946 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:54:59.252957 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:54:59.252961 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:54:59.253322 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:54:59.253587 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:54:59.253588 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:54:59.253588 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:54:59.253754 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:54:59.254108 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:54:59.254116 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:54:59.254316 (+ 200us) client_negotiation.cc:769] Sending connection context
0928 14:54:59.254930 (+ 614us) client_negotiation.cc:240] Negotiation successful
0928 14:54:59.255129 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260928 14:54:59.256083 32544 heartbeater.cc:347] Connected to a master server at 127.20.116.190:35569
I20260928 14:54:59.256201 32544 heartbeater.cc:464] Registering TS with master...
I20260928 14:54:59.256419 32544 heartbeater.cc:511] Master 127.20.116.190:35569 requested a full tablet report, sending...
I20260928 14:54:59.256896 32195 ts_manager.cc:194] Registered new tserver with Master: 918cd143ff1c42aeac93a4064b5272af (127.20.116.130:40825)
I20260928 14:54:59.257372 32195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:44483
WARNING: no policy specified for kudu/127.20.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
I20260928 14:54:59.285425 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.116.148:39267/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35569
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:34039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:54:59.411711 32550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:54:59.412036 32550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:54:59.412083 32550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:54:59.413941 32550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:54:59.414062 32550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:54:59.414155 32550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:54:59.416602 32550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:34039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.116.148:39267/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:54:59.417029 32550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:54:59.417383 32550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:54:59.424217 32555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:59.424252 32558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:54:59.424541 32556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:54:59.424957 32550 server_base.cc:1034] running on GCE node
I20260928 14:54:59.425206 32550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:54:59.425516 32550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:54:59.426762 32550 hybrid_clock.cc:648] HybridClock initialized: now 1790607299426743 us; error 34 us; skew 500 ppm
Sep 28 14:54:59 dist-test-slave-pxl6 krb5kdc[31718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607299, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:54:59.430822 32550 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:54:59.432339 32550 webserver.cc:460] Webserver started at http://127.20.116.131:35381/ using document root <none> and password file <none>
I20260928 14:54:59.432680 32550 fs_manager.cc:359] Metadata directory not provided
I20260928 14:54:59.432813 32550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:54:59.433022 32550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:54:59 dist-test-slave-pxl6 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607299, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for HTTP/127.20.116.148@KRBTEST.COM
I20260928 14:54:59.458884 32550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "c577c013235c4cea973d24d6694b6536"
format_stamp: "Formatted at 2026-09-28 14:54:59 on dist-test-slave-pxl6"
server_key: "84a1f265e21e0da46475571d7ff31b3f"
server_key_iv: "dd8cadb8d01cd93ce31012fd67d293e0"
server_key_version: "kuduclusterkey@0"
I20260928 14:54:59.459336 32550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "c577c013235c4cea973d24d6694b6536"
format_stamp: "Formatted at 2026-09-28 14:54:59 on dist-test-slave-pxl6"
server_key: "84a1f265e21e0da46475571d7ff31b3f"
server_key_iv: "dd8cadb8d01cd93ce31012fd67d293e0"
server_key_version: "kuduclusterkey@0"
I20260928 14:54:59.461355 32550 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:54:59.475891 32565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:59.476140 32550 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:54:59.476243 32550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "c577c013235c4cea973d24d6694b6536"
format_stamp: "Formatted at 2026-09-28 14:54:59 on dist-test-slave-pxl6"
server_key: "84a1f265e21e0da46475571d7ff31b3f"
server_key_iv: "dd8cadb8d01cd93ce31012fd67d293e0"
server_key_version: "kuduclusterkey@0"
I20260928 14:54:59.476351 32550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:54:59.500272 32550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:54:59.503083 32550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:54:59.503288 32550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:54:59.503588 32550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:54:59.504076 32550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:54:59.504143 32550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:59.504184 32550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:54:59.504266 32550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:54:59.512887 32550 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:34161
I20260928 14:54:59.513468 32550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:54:59.519155 32678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:34161 every 8 connection(s)
I20260928 14:54:59.523469 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 32550
I20260928 14:54:59.523612 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:54:59.523846 20946 external_mini_cluster.cc:1428] Setting key ae8bd84fc834278e4e5f7d3755d93115
Sep 28 14:54:59 dist-test-slave-pxl6 krb5kdc[31718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607299, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:54:59.530560 32408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:59.518427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:41281 (local address 127.20.116.190:35569)
0928 14:54:59.518555 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:59.518558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:59.518570 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:54:59.518600 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:59.518601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:59.518613 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:59.518681 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:59.519475 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.520224 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.521679 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.521969 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.522010 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:59.526892 (+ 4882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:54:59.526894 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:54:59.526895 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:54:59.526904 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:54:59.528793 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:54:59.529344 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:54:59.529346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:54:59.529346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:54:59.529407 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:54:59.529727 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:54:59.529728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:54:59.529728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:54:59.529876 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:54:59.529922 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:59.530345 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 14:54:59.530433 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 14:54:59.531420 32681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:59.514371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35569 (local address 127.20.116.131:41281)
0928 14:54:59.514628 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:59.514683 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:59.515339 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:59.518790 (+ 3451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:59.518795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:59.519064 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:54:59.519353 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:59.519359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.520347 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.520348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:59.521575 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:59.521579 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.522101 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.522102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:59.522240 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:59.524709 (+ 2469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:54:59.524721 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:54:59.526701 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:54:59.528894 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:54:59.528905 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:54:59.528909 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:54:59.529240 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:54:59.529483 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:54:59.529484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:54:59.529484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:54:59.529641 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:54:59.529973 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:54:59.529980 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:54:59.530166 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 14:54:59.530947 (+ 781us) client_negotiation.cc:240] Negotiation successful
0928 14:54:59.531150 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":93,"threads_started":1}
I20260928 14:54:59.532370 32679 heartbeater.cc:347] Connected to a master server at 127.20.116.190:35569
I20260928 14:54:59.532475 32679 heartbeater.cc:464] Registering TS with master...
I20260928 14:54:59.532680 32679 heartbeater.cc:511] Master 127.20.116.190:35569 requested a full tablet report, sending...
I20260928 14:54:59.533192 32195 ts_manager.cc:194] Registered new tserver with Master: c577c013235c4cea973d24d6694b6536 (127.20.116.131:34161)
I20260928 14:54:59.533754 32195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:41281
I20260928 14:54:59.535568 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:54:59.551128 32408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:59.540940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44998 (local address 127.20.116.190:35569)
0928 14:54:59.540988 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:59.540990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:59.541334 (+ 344us) server_negotiation.cc:407] Connection header received
0928 14:54:59.541444 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:59.541445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:59.541458 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:59.541490 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:54:59.543988 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.544806 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.545925 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.546226 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.546315 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:59.547934 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:54:59.547935 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:54:59.547936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:54:59.547944 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:54:59.549743 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:54:59.550177 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:54:59.550178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:54:59.550178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:54:59.550238 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:54:59.550455 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:54:59.550456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:54:59.550456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:54:59.550622 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:54:59.550707 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:59.550933 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 14:54:59.551016 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:54:59.553486 32195 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:54:59.555874 32195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:54:59.572701 32692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:59.565601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:37577 (local address 127.0.0.1:59886)
0928 14:54:59.567931 (+ 2330us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:59.567942 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:59.567999 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:59.568253 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:59.568254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:59.568255 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:54:59.568295 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:59.568297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.569868 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.569869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:59.571359 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:59.571362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.572410 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.572411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:59.572476 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:59.572480 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:54:59.572544 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 14:54:59.572615 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2312,"thread_start_us":56,"threads_started":1}
I20260928 14:54:59.573163 32697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:59.566024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59886 (local address 127.20.116.129:37577)
0928 14:54:59.567153 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:59.567157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:59.568061 (+ 904us) server_negotiation.cc:407] Connection header received
0928 14:54:59.568099 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:59.568102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:59.568118 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:59.568186 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:54:59.568369 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.569762 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.571456 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.572247 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.572321 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:59.572375 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:59.573044 (+ 669us) server_negotiation.cc:299] Negotiation successful
0928 14:54:59.573088 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1099,"thread_start_us":45,"threads_started":1}
I20260928 14:54:59.575446 32339 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0f9dcfd96f47a18fe785b8eb955fa2 (DEFAULT_TABLE table=test-table [id=b38c4cdd05304b7f8637f63b4d3c16dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:54:59.575902 32339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0f9dcfd96f47a18fe785b8eb955fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:54:59.578047 32694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:59.565393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:34161 (local address 127.0.0.1:46478)
0928 14:54:59.566806 (+ 1413us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:59.566854 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:59.566925 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:59.574030 (+ 7105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:59.574031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:59.574033 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:54:59.574069 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:59.574070 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.575385 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.575386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:59.576485 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:59.576488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.577824 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.577825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:59.577894 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:59.577899 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:54:59.577928 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 14:54:59.577969 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1379,"thread_start_us":78,"threads_started":1}
I20260928 14:54:59.578436 32695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:59.565679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46478 (local address 127.20.116.131:34161)
0928 14:54:59.573787 (+ 8108us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:59.573793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:59.573804 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:54:59.573848 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:59.573850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:59.573867 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:59.573943 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:54:59.574145 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.575282 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.576647 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.577727 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.578226 (+ 499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:59.578295 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:59.578321 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 14:54:59.578359 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7771,"thread_start_us":94,"threads_started":1}
I20260928 14:54:59.579095 32595 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0f9dcfd96f47a18fe785b8eb955fa2 (DEFAULT_TABLE table=test-table [id=b38c4cdd05304b7f8637f63b4d3c16dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:54:59.579428 32595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0f9dcfd96f47a18fe785b8eb955fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:54:59.582973 32693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:59.565395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:40825 (local address 127.0.0.1:47352)
0928 14:54:59.567201 (+ 1806us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:59.567210 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:59.567269 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:59.567729 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:59.567733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:59.567735 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:54:59.567815 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:59.567817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.569311 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.569313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:59.570960 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:59.570963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.572043 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.572044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:59.572123 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:59.572138 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 14:54:59.582837 (+ 10699us) client_negotiation.cc:240] Negotiation successful
0928 14:54:59.582891 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1791,"thread_start_us":96,"threads_started":1}
I20260928 14:54:59.583338 32696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:59.565864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47352 (local address 127.20.116.130:40825)
0928 14:54:59.567082 (+ 1218us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:59.567088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:59.567326 (+ 238us) server_negotiation.cc:407] Connection header received
0928 14:54:59.567370 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:59.567373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:59.567389 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:59.567456 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:54:59.567915 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.569200 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.571079 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.571850 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.571942 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:59.572006 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:59.583194 (+ 11188us) server_negotiation.cc:299] Negotiation successful
0928 14:54:59.583253 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1179,"thread_start_us":82,"threads_started":1}
I20260928 14:54:59.583529 32699 tablet_bootstrap.cc:492] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P c577c013235c4cea973d24d6694b6536: Bootstrap starting.
I20260928 14:54:59.584177 32478 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0f9dcfd96f47a18fe785b8eb955fa2 (DEFAULT_TABLE table=test-table [id=b38c4cdd05304b7f8637f63b4d3c16dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:54:59.584534 32478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0f9dcfd96f47a18fe785b8eb955fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:54:59.584611 32699 tablet_bootstrap.cc:654] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P c577c013235c4cea973d24d6694b6536: Neither blocks nor log segments found. Creating new log.
I20260928 14:54:59.584904 32699 log.cc:824] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P c577c013235c4cea973d24d6694b6536: Log is configured to *not* fsync() on all Append() calls
I20260928 14:54:59.585224 32698 tablet_bootstrap.cc:492] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4: Bootstrap starting.
I20260928 14:54:59.585884 32698 tablet_bootstrap.cc:654] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4: Neither blocks nor log segments found. Creating new log.
I20260928 14:54:59.586165 32698 log.cc:824] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:54:59.587014 32699 tablet_bootstrap.cc:492] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P c577c013235c4cea973d24d6694b6536: No bootstrap required, opened a new log
I20260928 14:54:59.587078 32699 ts_tablet_manager.cc:1397] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P c577c013235c4cea973d24d6694b6536: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 14:54:59.588227 32702 tablet_bootstrap.cc:492] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 918cd143ff1c42aeac93a4064b5272af: Bootstrap starting.
I20260928 14:54:59.589237 32702 tablet_bootstrap.cc:654] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 918cd143ff1c42aeac93a4064b5272af: Neither blocks nor log segments found. Creating new log.
I20260928 14:54:59.589340 32699 raft_consensus.cc:348] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P c577c013235c4cea973d24d6694b6536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918cd143ff1c42aeac93a4064b5272af" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40825 } } peers { permanent_uuid: "c577c013235c4cea973d24d6694b6536" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34161 } } peers { permanent_uuid: "4b607efee321476896c686337df987f4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 37577 } }
I20260928 14:54:59.589514 32699 raft_consensus.cc:374] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P c577c013235c4cea973d24d6694b6536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:54:59.589553 32699 raft_consensus.cc:729] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P c577c013235c4cea973d24d6694b6536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c577c013235c4cea973d24d6694b6536, State: Initialized, Role: FOLLOWER
I20260928 14:54:59.589675 32699 consensus_queue.cc:260] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P c577c013235c4cea973d24d6694b6536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918cd143ff1c42aeac93a4064b5272af" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40825 } } peers { permanent_uuid: "c577c013235c4cea973d24d6694b6536" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34161 } } peers { permanent_uuid: "4b607efee321476896c686337df987f4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 37577 } }
I20260928 14:54:59.589877 32702 log.cc:824] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 918cd143ff1c42aeac93a4064b5272af: Log is configured to *not* fsync() on all Append() calls
I20260928 14:54:59.589927 32699 ts_tablet_manager.cc:1428] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P c577c013235c4cea973d24d6694b6536: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:54:59.590065 32679 heartbeater.cc:503] Master 127.20.116.190:35569 was elected leader, sending a full tablet report...
I20260928 14:54:59.590636 32698 tablet_bootstrap.cc:492] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4: No bootstrap required, opened a new log
I20260928 14:54:59.590706 32698 ts_tablet_manager.cc:1397] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 14:54:59.592149 32702 tablet_bootstrap.cc:492] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 918cd143ff1c42aeac93a4064b5272af: No bootstrap required, opened a new log
I20260928 14:54:59.592272 32702 ts_tablet_manager.cc:1397] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 918cd143ff1c42aeac93a4064b5272af: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 14:54:59.593195 32698 raft_consensus.cc:348] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918cd143ff1c42aeac93a4064b5272af" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40825 } } peers { permanent_uuid: "c577c013235c4cea973d24d6694b6536" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34161 } } peers { permanent_uuid: "4b607efee321476896c686337df987f4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 37577 } }
I20260928 14:54:59.593367 32698 raft_consensus.cc:374] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:54:59.593407 32698 raft_consensus.cc:729] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b607efee321476896c686337df987f4, State: Initialized, Role: FOLLOWER
I20260928 14:54:59.593525 32698 consensus_queue.cc:260] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918cd143ff1c42aeac93a4064b5272af" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40825 } } peers { permanent_uuid: "c577c013235c4cea973d24d6694b6536" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34161 } } peers { permanent_uuid: "4b607efee321476896c686337df987f4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 37577 } }
I20260928 14:54:59.593796 32698 ts_tablet_manager.cc:1428] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:54:59.593924 32405 heartbeater.cc:503] Master 127.20.116.190:35569 was elected leader, sending a full tablet report...
I20260928 14:54:59.595343 32702 raft_consensus.cc:348] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 918cd143ff1c42aeac93a4064b5272af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918cd143ff1c42aeac93a4064b5272af" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40825 } } peers { permanent_uuid: "c577c013235c4cea973d24d6694b6536" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34161 } } peers { permanent_uuid: "4b607efee321476896c686337df987f4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 37577 } }
I20260928 14:54:59.595719 32702 raft_consensus.cc:374] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 918cd143ff1c42aeac93a4064b5272af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:54:59.595805 32702 raft_consensus.cc:729] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 918cd143ff1c42aeac93a4064b5272af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 918cd143ff1c42aeac93a4064b5272af, State: Initialized, Role: FOLLOWER
I20260928 14:54:59.596227 32702 consensus_queue.cc:260] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 918cd143ff1c42aeac93a4064b5272af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918cd143ff1c42aeac93a4064b5272af" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40825 } } peers { permanent_uuid: "c577c013235c4cea973d24d6694b6536" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34161 } } peers { permanent_uuid: "4b607efee321476896c686337df987f4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 37577 } }
I20260928 14:54:59.596607 32702 ts_tablet_manager.cc:1428] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 918cd143ff1c42aeac93a4064b5272af: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 14:54:59.596756 32544 heartbeater.cc:503] Master 127.20.116.190:35569 was elected leader, sending a full tablet report...
I20260928 14:54:59.596984 32706 raft_consensus.cc:482] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:54:59.597066 32706 raft_consensus.cc:504] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918cd143ff1c42aeac93a4064b5272af" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40825 } } peers { permanent_uuid: "c577c013235c4cea973d24d6694b6536" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34161 } } peers { permanent_uuid: "4b607efee321476896c686337df987f4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 37577 } }
I20260928 14:54:59.599835 32706 leader_election.cc:288] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 918cd143ff1c42aeac93a4064b5272af (127.20.116.130:40825), c577c013235c4cea973d24d6694b6536 (127.20.116.131:34161)
I20260928 14:54:59.601773 32696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:59.597671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:38225 (local address 127.20.116.130:40825)
0928 14:54:59.597718 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:59.597720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:59.598086 (+ 366us) server_negotiation.cc:407] Connection header received
0928 14:54:59.598185 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:59.598186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:59.598195 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:59.598225 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:54:59.598495 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.599266 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.600789 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.601383 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.601436 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:59.601445 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:59.601663 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 14:54:59.601704 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:54:59.602185 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:59.597472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:40825 (local address 127.20.116.129:38225)
0928 14:54:59.598018 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:59.598031 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:59.598113 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:59.598305 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:59.598305 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:59.598307 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:54:59.598389 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:59.598391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.599409 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.599410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:59.600671 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:59.600674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.601506 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.601507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:59.601578 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:59.601582 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:54:59.602050 (+ 468us) client_negotiation.cc:240] Negotiation successful
0928 14:54:59.602103 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":80,"threads_started":1}
I20260928 14:54:59.602686 32498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0f9dcfd96f47a18fe785b8eb955fa2" candidate_uuid: "4b607efee321476896c686337df987f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "918cd143ff1c42aeac93a4064b5272af" is_pre_election: true
I20260928 14:54:59.602847 32498 raft_consensus.cc:2454] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 918cd143ff1c42aeac93a4064b5272af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b607efee321476896c686337df987f4 in term 0.
I20260928 14:54:59.603211 32294 leader_election.cc:302] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b607efee321476896c686337df987f4, 918cd143ff1c42aeac93a4064b5272af; no voters:
I20260928 14:54:59.603442 32705 raft_consensus.cc:2781] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:54:59.603565 32705 raft_consensus.cc:482] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:54:59.604041 32705 raft_consensus.cc:3037] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:54:59.605080 32705 raft_consensus.cc:504] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918cd143ff1c42aeac93a4064b5272af" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40825 } } peers { permanent_uuid: "c577c013235c4cea973d24d6694b6536" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34161 } } peers { permanent_uuid: "4b607efee321476896c686337df987f4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 37577 } }
I20260928 14:54:59.605434 32498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0f9dcfd96f47a18fe785b8eb955fa2" candidate_uuid: "4b607efee321476896c686337df987f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "918cd143ff1c42aeac93a4064b5272af"
I20260928 14:54:59.605510 32498 raft_consensus.cc:3037] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 918cd143ff1c42aeac93a4064b5272af [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:54:59.605974 32695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:59.598017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:47571 (local address 127.20.116.131:34161)
0928 14:54:59.598053 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:59.598055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:59.600224 (+ 2169us) server_negotiation.cc:407] Connection header received
0928 14:54:59.600323 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:59.600324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:59.600336 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:59.600362 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:54:59.600560 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.601289 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.603042 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.603567 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.603683 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:59.603693 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:59.605857 (+ 2164us) server_negotiation.cc:299] Negotiation successful
0928 14:54:59.605910 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:54:59.606210 32708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:59.597856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:34161 (local address 127.20.116.129:47571)
0928 14:54:59.600153 (+ 2297us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:59.600165 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:54:59.600251 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:54:59.600422 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:54:59.600423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:54:59.600424 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:54:59.600491 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:59.600493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.601829 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.601830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:59.602938 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:54:59.602940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.605687 (+ 2747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.605688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:54:59.605767 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:59.605771 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:54:59.606075 (+ 304us) client_negotiation.cc:240] Negotiation successful
0928 14:54:59.606119 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2278,"thread_start_us":76,"threads_started":1}
I20260928 14:54:59.606400 32498 raft_consensus.cc:2454] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 918cd143ff1c42aeac93a4064b5272af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b607efee321476896c686337df987f4 in term 1.
I20260928 14:54:59.606503 32633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0f9dcfd96f47a18fe785b8eb955fa2" candidate_uuid: "4b607efee321476896c686337df987f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c577c013235c4cea973d24d6694b6536" is_pre_election: true
I20260928 14:54:59.606693 32633 raft_consensus.cc:2454] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P c577c013235c4cea973d24d6694b6536 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b607efee321476896c686337df987f4 in term 0.
I20260928 14:54:59.606855 32294 leader_election.cc:302] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b607efee321476896c686337df987f4, 918cd143ff1c42aeac93a4064b5272af; no voters:
I20260928 14:54:59.606947 32705 leader_election.cc:288] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4 [CANDIDATE]: Term 1 election: Requested vote from peers 918cd143ff1c42aeac93a4064b5272af (127.20.116.130:40825), c577c013235c4cea973d24d6694b6536 (127.20.116.131:34161)
I20260928 14:54:59.606984 32705 raft_consensus.cc:2781] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:54:59.607018 32705 raft_consensus.cc:686] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4 [term 1 LEADER]: Becoming Leader. State: Replica: 4b607efee321476896c686337df987f4, State: Running, Role: LEADER
I20260928 14:54:59.607098 32705 consensus_queue.cc:237] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918cd143ff1c42aeac93a4064b5272af" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40825 } } peers { permanent_uuid: "c577c013235c4cea973d24d6694b6536" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34161 } } peers { permanent_uuid: "4b607efee321476896c686337df987f4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 37577 } }
I20260928 14:54:59.607158 32633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0f9dcfd96f47a18fe785b8eb955fa2" candidate_uuid: "4b607efee321476896c686337df987f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c577c013235c4cea973d24d6694b6536"
I20260928 14:54:59.607227 32633 raft_consensus.cc:3037] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P c577c013235c4cea973d24d6694b6536 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:54:59.608065 32633 raft_consensus.cc:2454] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P c577c013235c4cea973d24d6694b6536 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b607efee321476896c686337df987f4 in term 1.
I20260928 14:54:59.608105 32195 catalog_manager.cc:5495] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b607efee321476896c686337df987f4 (127.20.116.129). New cstate: current_term: 1 leader_uuid: "4b607efee321476896c686337df987f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918cd143ff1c42aeac93a4064b5272af" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c577c013235c4cea973d24d6694b6536" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b607efee321476896c686337df987f4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 37577 } health_report { overall_health: HEALTHY } } }
I20260928 14:54:59.619297 32697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:59.615998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59892 (local address 127.20.116.129:37577)
0928 14:54:59.616042 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:54:59.616044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:54:59.616054 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:54:59.616115 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:54:59.616116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:54:59.616143 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:54:59.616213 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:54:59.616389 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.617116 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.618194 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:54:59.618465 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:54:59.618518 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:54:59.618746 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:54:59.618881 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:54:59.618949 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:54:59.619171 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 14:54:59.619193 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:54:59.621567 32633 raft_consensus.cc:1261] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P c577c013235c4cea973d24d6694b6536 [term 1 FOLLOWER]: Refusing update from remote peer 4b607efee321476896c686337df987f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:54:59.621613 32498 raft_consensus.cc:1261] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 918cd143ff1c42aeac93a4064b5272af [term 1 FOLLOWER]: Refusing update from remote peer 4b607efee321476896c686337df987f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:54:59.621958 32706 consensus_queue.cc:1035] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "918cd143ff1c42aeac93a4064b5272af" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 40825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:54:59.622126 32706 consensus_queue.cc:1035] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 4b607efee321476896c686337df987f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c577c013235c4cea973d24d6694b6536" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:54:59.626242 32712 mvcc.cc:205] Tried to move back new op lower bound from 7334327499247083520 to 7334327499191824384. Current Snapshot: MvccSnapshot[applied={T|T < 7334327499247083520}]
I20260928 14:54:59.626510 32713 mvcc.cc:205] Tried to move back new op lower bound from 7334327499247083520 to 7334327499191824384. Current Snapshot: MvccSnapshot[applied={T|T < 7334327499247083520}]
I20260928 14:54:59.627328 32714 mvcc.cc:205] Tried to move back new op lower bound from 7334327499247083520 to 7334327499191824384. Current Snapshot: MvccSnapshot[applied={T|T < 7334327499247083520 or (T in {7334327499247083520})}]
I20260928 14:54:59.630067 32194 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:54:59.630198 32194 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:54:59.631145 32194 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54bdb44f4b1f4be8bacdb7c49fdd650b: Sending DeleteTablet for 3 replicas of tablet 6b0f9dcfd96f47a18fe785b8eb955fa2
I20260928 14:54:59.631557 32478 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0f9dcfd96f47a18fe785b8eb955fa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:54:59 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:47352
I20260928 14:54:59.631861 32339 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0f9dcfd96f47a18fe785b8eb955fa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:54:59 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:59886
I20260928 14:54:59.632215 32595 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0f9dcfd96f47a18fe785b8eb955fa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:54:59 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:46478
I20260928 14:54:59.632876 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 32276
I20260928 14:54:59.637224 32724 tablet_replica.cc:330] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P c577c013235c4cea973d24d6694b6536: stopping tablet replica
I20260928 14:54:59.637377 32724 raft_consensus.cc:2229] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P c577c013235c4cea973d24d6694b6536 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:54:59.637514 32724 raft_consensus.cc:2258] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P c577c013235c4cea973d24d6694b6536 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:54:59.637735 32722 tablet_replica.cc:330] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 918cd143ff1c42aeac93a4064b5272af: stopping tablet replica
I20260928 14:54:59.638038 32724 ts_tablet_manager.cc:1905] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P c577c013235c4cea973d24d6694b6536: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:54:59.639894 32724 ts_tablet_manager.cc:1918] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P c577c013235c4cea973d24d6694b6536: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:54:59.640034 32724 log.cc:1196] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P c577c013235c4cea973d24d6694b6536: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/6b0f9dcfd96f47a18fe785b8eb955fa2
I20260928 14:54:59.640676 32724 ts_tablet_manager.cc:1939] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P c577c013235c4cea973d24d6694b6536: Deleting consensus metadata
I20260928 14:54:59.641193 32180 catalog_manager.cc:4841] TS c577c013235c4cea973d24d6694b6536 (127.20.116.131:34161): tablet 6b0f9dcfd96f47a18fe785b8eb955fa2 (table test-table [id=b38c4cdd05304b7f8637f63b4d3c16dd]) successfully deleted
I20260928 14:54:59.641436 32722 raft_consensus.cc:2229] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 918cd143ff1c42aeac93a4064b5272af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:54:59.641669 32722 raft_consensus.cc:2258] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 918cd143ff1c42aeac93a4064b5272af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:54:59.642221 32722 ts_tablet_manager.cc:1905] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 918cd143ff1c42aeac93a4064b5272af: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 14:54:59.643334 32181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.116.129:37577 (error 108)
I20260928 14:54:59.643832 32722 ts_tablet_manager.cc:1918] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 918cd143ff1c42aeac93a4064b5272af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:54:59.643935 32722 log.cc:1196] T 6b0f9dcfd96f47a18fe785b8eb955fa2 P 918cd143ff1c42aeac93a4064b5272af: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/6b0f9dcfd96f47a18fe785b8eb955fa2
I20260928 14:54:59.644028 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 32413
I20260928 14:54:59.647346 32693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:59.647199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:37577 (local address 127.0.0.1:59904)
0928 14:54:59.647262 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:59.647304 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.129:37577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 14:54:59.647580 32181 catalog_manager.cc:4584] TS 4b607efee321476896c686337df987f4 (127.20.116.129:37577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b0f9dcfd96f47a18fe785b8eb955fa2: Network error: Client connection negotiation failed: client connection to 127.20.116.129:37577: connect: Connection refused (error 111)
I20260928 14:54:59.651700 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 32550
I20260928 14:54:59.652765 32693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:54:59.652629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:40825 (local address 127.0.0.1:47364)
0928 14:54:59.652682 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 14:54:59.652722 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.130:40825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 14:54:59.658466 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 32163
I20260928 14:54:59.664448 20946 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 14:55:00.009698 20946 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 14:55:00.009821 20946 mini_ranger.cc:66] Stopping Ranger...
I20260928 14:55:00.354918 20946 mini_ranger.cc:68] Stopped Ranger
2026-09-28 14:55:00.355 UTC [31748] LOG: received smart shutdown request
2026-09-28 14:55:00.359 UTC [31748] LOG: background worker "logical replication launcher" (PID 31757) exited with exit code 1
2026-09-28 14:55:00.359 UTC [31752] LOG: shutting down
2026-09-28 14:55:00.552 UTC [31748] LOG: database system is shut down
2026-09-28T14:55:00Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49198 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:55:00 dist-test-slave-pxl6 krb5kdc[32734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:55:00 dist-test-slave-pxl6 krb5kdc[32734](info): set up 2 sockets
Sep 28 14:55:00 dist-test-slave-pxl6 krb5kdc[32734](info): commencing operation
krb5kdc: starting...
W20260928 14:55:01.699479 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:55:01 dist-test-slave-pxl6 krb5kdc[32734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:55:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:55:01Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/rangeradmin_127.20.116.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/rangeradmin_127.20.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/rangerlookup_127.20.116.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/rangerlookup_127.20.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/HTTP_127.20.116.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/HTTP_127.20.116.148@KRBTEST.COM.keytab.
I20260928 14:55:01.881048 20946 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHOCNdC/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/postgres -l logfile start
2026-09-28 14:55:04.646 UTC [32764] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 14:55:04.646 UTC [32764] LOG: listening on IPv4 address "127.20.116.148", port 55171
2026-09-28 14:55:04.650 UTC [32764] LOG: listening on Unix socket "/tmp/.s.PGSQL.55171"
2026-09-28 14:55:04.672 UTC [32767] LOG: database system was shut down at 2026-09-28 14:55:02 UTC
2026-09-28 14:55:04.676 UTC [32764] LOG: database system is ready to accept connections
I20260928 14:55:05.631615 20946 mini_postgres.cc:96] Postgres bound to 55171
2026-09-28 14:55:05.638 UTC [307] FATAL: database "slave" does not exist
127.20.116.148:55171 - accepting connections
I20260928 14:55:05.639431 20946 mini_ranger.cc:161] Starting Ranger...
I20260928 14:55:05.654714 20946 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 14:55:06.210083 20946 mini_ranger.cc:87] Created ranger Postgres database
I20260928 14:55:06.210155 20946 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ranger-admin
2026-09-28 14:55:06,508 [I] DB FLAVOR :POSTGRES
2026-09-28 14:55:06,508 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:55:06,508 [I] Checking connection..
2026-09-28 14:55:06,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 14:55:06,742 [I] Checking connection passed.
2026-09-28 14:55:06,742 [I] --------- Verifying version history table ---------
2026-09-28 14:55:06,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:55:07,022 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:55:07,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:55:07,264 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:55:07,265 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 14:55:07,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 14:55:07.519 UTC [371] WARNING: there is no transaction in progress
2026-09-28 14:55:07,525 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 14:55:07,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:55:07,767 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 14:55:07,767 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:55:07,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 14:55:08,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 14:55:08,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pxl6.c.gcp-upstream.internal','N') ;"
2026-09-28 14:55:08,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:55:08,740 [I] Table x_portal_user does not exist in database ranger
2026-09-28 14:55:08,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:55:08,995 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 14:55:08,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:55:09,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:55:09,521 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 14:55:09,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 14:55:10.584 UTC [480] WARNING: there is no transaction in progress
2026-09-28 14:55:10.601 UTC [480] WARNING: there is no transaction in progress
2026-09-28 14:55:10.619 UTC [480] WARNING: there is no transaction in progress
2026-09-28 14:55:10.636 UTC [480] WARNING: there is no transaction in progress
2026-09-28 14:55:10.656 UTC [480] WARNING: there is no transaction in progress
2026-09-28 14:55:10.680 UTC [480] WARNING: there is no transaction in progress
2026-09-28 14:55:10.749 UTC [480] WARNING: there is no transaction in progress
2026-09-28 14:55:10.757 UTC [480] WARNING: there is no transaction in progress
2026-09-28 14:55:10.773 UTC [480] WARNING: there is no transaction in progress
2026-09-28 14:55:10.784 UTC [480] WARNING: there is no transaction in progress
2026-09-28 14:55:10.795 UTC [480] WARNING: there is no transaction in progress
2026-09-28 14:55:10.810 UTC [480] WARNING: there is no transaction in progress
2026-09-28 14:55:10.820 UTC [480] WARNING: there is no transaction in progress
2026-09-28 14:55:10.826 UTC [480] WARNING: there is no transaction in progress
2026-09-28 14:55:10.833 UTC [480] WARNING: there is no transaction in progress
2026-09-28 14:55:10.840 UTC [480] WARNING: there is no transaction in progress
2026-09-28 14:55:11,552 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 14:55:11,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 14:55:11,789 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 14:55:11,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:55:12,060 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 14:55:12,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:55:12,317 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 14:55:12,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:55:12,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:55:12,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pxl6.c.gcp-upstream.internal';"
2026-09-28 14:55:13,052 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 14:55:13,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 14:55:13,303 [I] DB_PATCHES have already been applied
I20260928 14:55:13.309484 20946 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHOCNdC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 14:55:13.309554 20946 mini_ranger.cc:193] Using host: 127.20.116.148
I20260928 14:55:13.312932 20946 mini_ranger.cc:227] Ranger admin URL: http://127.20.116.148:56791
Sep 28, 2026 2:55:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:55:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:55:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:55:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:55:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:55:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:55:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 14:55:14.414157 20946 mini_ranger.cc:160] Time spent starting Ranger: real 8.775s user 0.004s sys 0.005s
Sep 28, 2026 2:55:35 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:55:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:55:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:55:35 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:55:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:55:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:55:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:55:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:55:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:55:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:55:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:55:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:55:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:55:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:55:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:55:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:55:35 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:55:36 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 14:55:36.658618 20946 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/rangerkms_127.20.116.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/rangerkms_127.20.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.116.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.116.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.116.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/HTTP_127.20.116.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.116.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/HTTP_127.20.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 14:55:36 dist-test-slave-pxl6 krb5kdc[32734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607336, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 14:55:36.744851 20946 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 14:55:36.755188 20946 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 14:55:37.718230 20946 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 14:55:37.718302 20946 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ranger-kms
I20260928 14:55:37.718315 20946 mini_ranger_kms.cc:227] Using postgres at 127.20.116.148:55171
2026-09-28 14:55:37,890 [I] DB FLAVOR :POSTGRES
2026-09-28 14:55:37,890 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:55:37,890 [I] Checking connection
2026-09-28 14:55:37,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 14:55:38,130 [I] connection success
2026-09-28 14:55:38,131 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 14:55:38,131 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 14:55:38,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 14:55:38,384 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 14:55:38,384 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:55:38,384 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 14:55:38,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.116.148:55171/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 14:55:38,657 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 14:55:39.117566 20946 mini_ranger_kms.cc:310] Created kms service
I20260928 14:55:39.288776 20946 mini_ranger_kms.cc:326] Created kudu user
I20260928 14:55:39.333747 20946 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 14:55:39 dist-test-slave-pxl6 krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607336, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.116.148@KRBTEST.COM
I20260928 14:55:39.765981 20946 mini_ranger_kms.cc:384] Added ranger policy
I20260928 14:55:39.766103 20946 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHOCNdC/build/release/bin/postgresql.jar:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHOCNdC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 14:55:39.766124 20946 mini_ranger_kms.cc:240] Using host: 127.20.116.148
I20260928 14:55:39.769645 20946 mini_ranger_kms.cc:276] Ranger KMS PID: 660
I20260928 14:55:39.769742 20946 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.116.148:50385
Sep 28, 2026 2:55:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:55:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:55:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:55:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:55:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:55:41 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 14:55:42.051899 20946 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.307s user 0.003s sys 0.152s
I20260928 14:55:42.052014 20946 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 14:55:42.052065 20946 mini_ranger_kms.cc:401] 127.20.116.148:50385/kms/v1/keys
Sep 28 14:55:42 dist-test-slave-pxl6 krb5kdc[32734](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607342, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.116.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 14:55:42 dist-test-slave-pxl6 krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607342, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.116.148@KRBTEST.COM for HTTP/127.20.116.148@KRBTEST.COM
Sep 28 14:55:42 dist-test-slave-pxl6 krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607342, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.116.148@KRBTEST.COM for HTTP/127.20.116.148@KRBTEST.COM
Sep 28 14:55:45 dist-test-slave-pxl6 krb5kdc[32734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:55:45.306617 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:43699
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:43699
--ranger_config_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.116.148:50385/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:55:45.391736 712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:45.391948 712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:45.391973 712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:45.393226 712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:55:45.393276 712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:45.393290 712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:45.393303 712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:55:45.393316 712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:55:45.395018 712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.116.148:50385/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:43699
--ranger_config_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:43699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:45.395277 712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:45.395520 712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:45.400638 720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:45.400631 718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:45.400866 712 server_base.cc:1034] running on GCE node
W20260928 14:55:45.400650 717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:45.401301 712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:45.401657 712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:45.402824 712 hybrid_clock.cc:648] HybridClock initialized: now 1790607345402803 us; error 34 us; skew 500 ppm
Sep 28 14:55:45 dist-test-slave-pxl6 krb5kdc[32734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607345, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:45.410012 712 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:55:45.410632 712 webserver.cc:460] Webserver started at http://127.20.116.190:40445/ using document root <none> and password file <none>
I20260928 14:55:45.410867 712 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:45.410924 712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:55:45.411080 712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:55:45 dist-test-slave-pxl6 krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607345, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for HTTP/127.20.116.148@KRBTEST.COM
I20260928 14:55:45.470428 712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "902fed5c3300416dbcc2fc1f3aa94320"
format_stamp: "Formatted at 2026-09-28 14:55:45 on dist-test-slave-pxl6"
server_key: "ea72f51dedbeca901d6cc17c4d465ade"
server_key_iv: "3ea5a7b4d7543fd426dde7866e0e7637"
server_key_version: "kuduclusterkey@0"
I20260928 14:55:45.470942 712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "902fed5c3300416dbcc2fc1f3aa94320"
format_stamp: "Formatted at 2026-09-28 14:55:45 on dist-test-slave-pxl6"
server_key: "ea72f51dedbeca901d6cc17c4d465ade"
server_key_iv: "3ea5a7b4d7543fd426dde7866e0e7637"
server_key_version: "kuduclusterkey@0"
I20260928 14:55:45.472798 712 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:55:45.487212 727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:45.487977 712 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:55:45.488098 712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "902fed5c3300416dbcc2fc1f3aa94320"
format_stamp: "Formatted at 2026-09-28 14:55:45 on dist-test-slave-pxl6"
server_key: "ea72f51dedbeca901d6cc17c4d465ade"
server_key_iv: "3ea5a7b4d7543fd426dde7866e0e7637"
server_key_version: "kuduclusterkey@0"
I20260928 14:55:45.488200 712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:45.508651 712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:45.511039 712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:45.511227 712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:45.516732 712 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:43699
I20260928 14:55:45.516772 779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:43699 every 8 connection(s)
I20260928 14:55:45.517239 712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:55:45.518020 780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:55:45.520424 780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320: Bootstrap starting.
I20260928 14:55:45.521075 780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320: Neither blocks nor log segments found. Creating new log.
I20260928 14:55:45.521369 780 log.cc:824] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320: Log is configured to *not* fsync() on all Append() calls
I20260928 14:55:45.522074 780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320: No bootstrap required, opened a new log
I20260928 14:55:45.523365 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 712
I20260928 14:55:45.523471 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:55:45.523676 20946 external_mini_cluster.cc:1428] Setting key c058df37c794e0ba3746eb56676c70f4
I20260928 14:55:45.524881 780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902fed5c3300416dbcc2fc1f3aa94320" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43699 } }
I20260928 14:55:45.525103 780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:55:45.525179 780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 902fed5c3300416dbcc2fc1f3aa94320, State: Initialized, Role: FOLLOWER
I20260928 14:55:45.525360 780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902fed5c3300416dbcc2fc1f3aa94320" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43699 } }
I20260928 14:55:45.525458 780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:55:45.525512 780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:55:45.525584 780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:55:45.526283 780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902fed5c3300416dbcc2fc1f3aa94320" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43699 } }
I20260928 14:55:45.526396 780 leader_election.cc:302] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 902fed5c3300416dbcc2fc1f3aa94320; no voters:
I20260928 14:55:45.526628 780 leader_election.cc:288] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:55:45.526683 785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:55:45.526923 785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [term 1 LEADER]: Becoming Leader. State: Replica: 902fed5c3300416dbcc2fc1f3aa94320, State: Running, Role: LEADER
I20260928 14:55:45.527007 780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:55:45.527117 785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902fed5c3300416dbcc2fc1f3aa94320" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43699 } }
I20260928 14:55:45.527554 785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "902fed5c3300416dbcc2fc1f3aa94320" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902fed5c3300416dbcc2fc1f3aa94320" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43699 } } }
I20260928 14:55:45.527747 786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 902fed5c3300416dbcc2fc1f3aa94320. Latest consensus state: current_term: 1 leader_uuid: "902fed5c3300416dbcc2fc1f3aa94320" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902fed5c3300416dbcc2fc1f3aa94320" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43699 } } }
I20260928 14:55:45.528851 786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [sys.catalog]: This master's current role is: LEADER
I20260928 14:55:45.528532 780 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 14:55:45.529433 785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [sys.catalog]: This master's current role is: LEADER
Sep 28 14:55:45 dist-test-slave-pxl6 krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:55:45.532481 800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:55:45.533439 800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:55:45.538345 784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:45.524287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44136 (local address 127.20.116.190:43699)
0928 14:55:45.524508 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:45.524521 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:45.524555 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:55:45.524843 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:45.524852 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:45.524922 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:45.525082 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:45.526066 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:45.527586 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:45.528761 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:45.529399 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:45.529535 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:45.533910 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:45.533934 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:45.533935 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:45.533958 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:45.536136 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:45.536546 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:45.536549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:45.536550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:45.536685 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:45.536909 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:45.536910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:45.536910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:45.537245 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:45.537404 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:45.537932 (+ 528us) server_negotiation.cc:299] Negotiation successful
0928 14:55:45.538102 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
I20260928 14:55:45.540774 800 catalog_manager.cc:1347] Generated new cluster ID: 5278728b318d4b8292cd12137f1c1167
I20260928 14:55:45.540827 800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:55:45.550436 800 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:55:45.551395 800 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:55:45.556317 800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320: Generated new TSK 0
I20260928 14:55:45.556602 800 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
I20260928 14:55:46.305058 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.116.148:50385/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:43699
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:55:46.408682 826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:46.408946 826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:46.408988 826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:46.410193 826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:46.410265 826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:46.410311 826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:55:46.411892 826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.116.148:50385/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:43699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:46.412165 826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:46.412402 826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:46.419428 832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:46.419529 834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:46.419744 831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:46.420228 826 server_base.cc:1034] running on GCE node
I20260928 14:55:46.420459 826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:46.420773 826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:46.422605 826 hybrid_clock.cc:648] HybridClock initialized: now 1790607346422638 us; error 90 us; skew 500 ppm
Sep 28 14:55:46 dist-test-slave-pxl6 krb5kdc[32734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607346, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:46.425940 826 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:55:46.426575 826 webserver.cc:460] Webserver started at http://127.20.116.129:35519/ using document root <none> and password file <none>
I20260928 14:55:46.426870 826 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:46.426946 826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:55:46.427098 826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:55:46 dist-test-slave-pxl6 krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607346, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for HTTP/127.20.116.148@KRBTEST.COM
I20260928 14:55:46.442315 826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "7647f0c709974cfa854d509ee77ae2ff"
format_stamp: "Formatted at 2026-09-28 14:55:46 on dist-test-slave-pxl6"
server_key: "12733cac10f2386ffe8181825a724d7d"
server_key_iv: "79e4ae69a4512cdb45fe50867ccb398a"
server_key_version: "kuduclusterkey@0"
I20260928 14:55:46.442785 826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "7647f0c709974cfa854d509ee77ae2ff"
format_stamp: "Formatted at 2026-09-28 14:55:46 on dist-test-slave-pxl6"
server_key: "12733cac10f2386ffe8181825a724d7d"
server_key_iv: "79e4ae69a4512cdb45fe50867ccb398a"
server_key_version: "kuduclusterkey@0"
I20260928 14:55:46.444247 826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:55:46.461722 843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:46.461977 826 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260928 14:55:46.462075 826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "7647f0c709974cfa854d509ee77ae2ff"
format_stamp: "Formatted at 2026-09-28 14:55:46 on dist-test-slave-pxl6"
server_key: "12733cac10f2386ffe8181825a724d7d"
server_key_iv: "79e4ae69a4512cdb45fe50867ccb398a"
server_key_version: "kuduclusterkey@0"
I20260928 14:55:46.462170 826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:46.472892 826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:46.475107 826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:46.475265 826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:46.475483 826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:55:46.475854 826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:55:46.475904 826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:46.475947 826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:55:46.475973 826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:46.483489 826 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:40603
I20260928 14:55:46.483875 826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:55:46.484447 957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:40603 every 8 connection(s)
Sep 28 14:55:46 dist-test-slave-pxl6 krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607346, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:55:46.492156 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 826
I20260928 14:55:46.492259 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:55:46.492471 20946 external_mini_cluster.cc:1428] Setting key 385916863ad81245d4ababa870586757
I20260928 14:55:46.495369 960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:46.484710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43699 (local address 127.20.116.129:52067)
0928 14:55:46.484891 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:46.484935 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:46.485516 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:46.485794 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:46.485798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:46.485996 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:46.486198 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:46.486203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:46.487022 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:46.487023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:46.488280 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:46.488284 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:46.488700 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:46.488701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:46.488843 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:46.489440 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:46.489454 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:46.490829 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:46.493603 (+ 2774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:46.493612 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:46.493615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:46.493938 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:46.494167 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:46.494168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:46.494168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:46.494316 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:46.494661 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:46.494667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:46.494848 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 14:55:46.494963 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 14:55:46.495122 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":101,"threads_started":1}
I20260928 14:55:46.495999 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:46.485019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:52067 (local address 127.20.116.190:43699)
0928 14:55:46.485175 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:46.485177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:46.485189 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:55:46.485626 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:46.485627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:46.485642 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:46.485710 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:46.486293 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:46.486925 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:46.488382 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:46.488602 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:46.488678 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:46.490932 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:46.490933 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:46.490933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:46.490941 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:46.493502 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:46.494029 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:46.494031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:46.494031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:46.494097 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:46.494389 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:46.494390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:46.494390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:46.494560 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:46.494623 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:46.495805 (+ 1182us) server_negotiation.cc:299] Negotiation successful
0928 14:55:46.495901 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260928 14:55:46.500970 958 heartbeater.cc:347] Connected to a master server at 127.20.116.190:43699
I20260928 14:55:46.501081 958 heartbeater.cc:464] Registering TS with master...
I20260928 14:55:46.501293 958 heartbeater.cc:511] Master 127.20.116.190:43699 requested a full tablet report, sending...
I20260928 14:55:46.501978 743 ts_manager.cc:194] Registered new tserver with Master: 7647f0c709974cfa854d509ee77ae2ff (127.20.116.129:40603)
I20260928 14:55:46.502948 743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:52067
WARNING: no policy specified for kudu/127.20.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
I20260928 14:55:46.533452 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.116.148:50385/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:43699
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:55:46 dist-test-slave-pxl6 krb5kdc[32734](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607346, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 14:55:46.636989 966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:46.637370 966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:46.637454 966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:46.639304 966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:46.639422 966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:46.639508 966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:55:46.641786 966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.116.148:50385/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:43699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:46.642124 966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:46.642445 966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:46.649408 971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:46.649362 974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:46.649356 972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:46.650727 966 server_base.cc:1034] running on GCE node
I20260928 14:55:46.650995 966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:46.651293 966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:46.652482 966 hybrid_clock.cc:648] HybridClock initialized: now 1790607346652462 us; error 35 us; skew 500 ppm
Sep 28 14:55:46 dist-test-slave-pxl6 krb5kdc[32734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607346, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:46.655791 966 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:55:46.656339 966 webserver.cc:460] Webserver started at http://127.20.116.130:40437/ using document root <none> and password file <none>
I20260928 14:55:46.656612 966 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:46.656673 966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:55:46.656804 966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:55:46 dist-test-slave-pxl6 krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607346, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for HTTP/127.20.116.148@KRBTEST.COM
I20260928 14:55:46.675091 966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "212b9b577ee04d11a86a9f319911ba39"
format_stamp: "Formatted at 2026-09-28 14:55:46 on dist-test-slave-pxl6"
server_key: "af205b1c13755d1a6d0c72dfe78871b6"
server_key_iv: "f3cde576d7ce5d54c95f5929733808ae"
server_key_version: "kuduclusterkey@0"
I20260928 14:55:46.675560 966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "212b9b577ee04d11a86a9f319911ba39"
format_stamp: "Formatted at 2026-09-28 14:55:46 on dist-test-slave-pxl6"
server_key: "af205b1c13755d1a6d0c72dfe78871b6"
server_key_iv: "f3cde576d7ce5d54c95f5929733808ae"
server_key_version: "kuduclusterkey@0"
I20260928 14:55:46.678939 966 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 14:55:46.694160 982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:46.694378 966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:55:46.694484 966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "212b9b577ee04d11a86a9f319911ba39"
format_stamp: "Formatted at 2026-09-28 14:55:46 on dist-test-slave-pxl6"
server_key: "af205b1c13755d1a6d0c72dfe78871b6"
server_key_iv: "f3cde576d7ce5d54c95f5929733808ae"
server_key_version: "kuduclusterkey@0"
I20260928 14:55:46.694618 966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:46.736474 966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:46.738719 966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:46.738883 966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:46.739161 966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:55:46.739557 966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:55:46.739606 966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:46.739647 966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:55:46.739672 966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:46.746332 966 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:35027
I20260928 14:55:46.746628 1097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:35027 every 8 connection(s)
I20260928 14:55:46.746855 966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:55:46 dist-test-slave-pxl6 krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607346, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:55:46.754720 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 966
I20260928 14:55:46.755098 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:55:46.755352 20946 external_mini_cluster.cc:1428] Setting key 850a7136395f7730472658f5cda25b9c
I20260928 14:55:46.757437 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:46.747964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:60609 (local address 127.20.116.190:43699)
0928 14:55:46.748020 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:46.748022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:46.748357 (+ 335us) server_negotiation.cc:407] Connection header received
0928 14:55:46.748892 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:46.748894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:46.748911 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:46.748967 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:46.749583 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:46.750099 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:46.751292 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:46.751529 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:46.751616 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:46.753956 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:46.753957 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:46.753957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:46.753965 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:46.755727 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:46.756227 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:46.756228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:46.756228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:46.756294 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:46.756603 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:46.756604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:46.756604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:46.756771 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:46.756844 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:46.757189 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 14:55:46.757290 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:55:46.757566 1100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:46.747641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43699 (local address 127.20.116.130:60609)
0928 14:55:46.748235 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:46.748270 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:46.748787 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:46.749042 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:46.749047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:46.749292 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:46.749489 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:46.749495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:46.750203 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:46.750204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:46.751186 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:46.751189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:46.751637 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:46.751638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:46.751774 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:46.752312 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:46.752321 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:46.753856 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:46.755812 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:46.755820 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:46.755822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:46.756144 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:46.756368 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:46.756368 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:46.756369 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:46.756520 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:46.756849 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:46.756856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:46.757032 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 14:55:46.757158 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 14:55:46.757321 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":96,"threads_started":1}
I20260928 14:55:46.758265 1098 heartbeater.cc:347] Connected to a master server at 127.20.116.190:43699
I20260928 14:55:46.758414 1098 heartbeater.cc:464] Registering TS with master...
I20260928 14:55:46.758697 1098 heartbeater.cc:511] Master 127.20.116.190:43699 requested a full tablet report, sending...
I20260928 14:55:46.759117 743 ts_manager.cc:194] Registered new tserver with Master: 212b9b577ee04d11a86a9f319911ba39 (127.20.116.130:35027)
I20260928 14:55:46.759622 743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:60609
WARNING: no policy specified for kudu/127.20.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
I20260928 14:55:46.785075 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.116.148:50385/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:43699
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:55:46.882395 1103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:46.882815 1103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:46.882893 1103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:46.884459 1103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:46.884534 1103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:46.884582 1103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:55:46.886175 1103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.116.148:50385/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:43699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:46.886457 1103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:46.886739 1103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:46.892027 1111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:46.892098 1109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:46.892395 1103 server_base.cc:1034] running on GCE node
W20260928 14:55:46.892242 1108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:46.892771 1103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:46.893046 1103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:46.894204 1103 hybrid_clock.cc:648] HybridClock initialized: now 1790607346894181 us; error 35 us; skew 500 ppm
Sep 28 14:55:46 dist-test-slave-pxl6 krb5kdc[32734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607346, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:46.897094 1103 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:55:46.897696 1103 webserver.cc:460] Webserver started at http://127.20.116.131:39547/ using document root <none> and password file <none>
I20260928 14:55:46.897970 1103 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:46.898031 1103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:55:46.898182 1103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:55:46 dist-test-slave-pxl6 krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607346, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for HTTP/127.20.116.148@KRBTEST.COM
I20260928 14:55:46.911015 1103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "595b4cbd51a74962a8ec4088e07fd2bb"
format_stamp: "Formatted at 2026-09-28 14:55:46 on dist-test-slave-pxl6"
server_key: "cb49a14ab60acd3c8f21f122cdc77adf"
server_key_iv: "9e0e33349c332d5fb2cdb102408e2603"
server_key_version: "kuduclusterkey@0"
I20260928 14:55:46.911484 1103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "595b4cbd51a74962a8ec4088e07fd2bb"
format_stamp: "Formatted at 2026-09-28 14:55:46 on dist-test-slave-pxl6"
server_key: "cb49a14ab60acd3c8f21f122cdc77adf"
server_key_iv: "9e0e33349c332d5fb2cdb102408e2603"
server_key_version: "kuduclusterkey@0"
I20260928 14:55:46.914232 1103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 14:55:46.948063 1118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:46.948304 1103 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.000s sys 0.000s
I20260928 14:55:46.948423 1103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "595b4cbd51a74962a8ec4088e07fd2bb"
format_stamp: "Formatted at 2026-09-28 14:55:46 on dist-test-slave-pxl6"
server_key: "cb49a14ab60acd3c8f21f122cdc77adf"
server_key_iv: "9e0e33349c332d5fb2cdb102408e2603"
server_key_version: "kuduclusterkey@0"
I20260928 14:55:46.948545 1103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:46.990368 1103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:46.993258 1103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:46.993495 1103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:46.993861 1103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:55:46.994305 1103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:55:46.994395 1103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:46.994468 1103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:55:46.994561 1103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:47.001711 1103 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:37953
I20260928 14:55:47.002246 1103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:55:47.003422 1232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:37953 every 8 connection(s)
Sep 28 14:55:47 dist-test-slave-pxl6 krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607346, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:55:47.011252 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 1103
I20260928 14:55:47.011363 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:55:47.011600 20946 external_mini_cluster.cc:1428] Setting key e1638b609c20e716a50bdb08e7ed50f5
I20260928 14:55:47.014747 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:47.003684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:37871 (local address 127.20.116.190:43699)
0928 14:55:47.003734 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:47.003736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:47.004015 (+ 279us) server_negotiation.cc:407] Connection header received
0928 14:55:47.004753 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:47.004755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:47.004769 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:47.004846 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:47.005671 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:47.006424 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:47.008023 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:47.008332 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:47.008376 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:47.011205 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:47.011206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:47.011207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:47.011216 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:47.012992 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:47.013511 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:47.013512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:47.013513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:47.013597 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:47.013920 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:47.013921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:47.013922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:47.014079 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:47.014158 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:47.014507 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 14:55:47.014649 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:55:47.014999 1235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:47.003580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43699 (local address 127.20.116.131:37871)
0928 14:55:47.003844 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:47.003893 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:47.004618 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:47.004945 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:47.004949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:47.005239 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:47.005558 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:47.005565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:47.006592 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:47.006593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:47.007912 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:47.007915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:47.008484 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:47.008485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:47.008640 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:47.009306 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:47.009318 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:47.011063 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:47.013096 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:47.013104 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:47.013108 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:47.013423 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:47.013680 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:47.013681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:47.013681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:47.013838 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:47.014160 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:47.014167 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:47.014349 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 14:55:47.014475 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 14:55:47.014685 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":87,"threads_started":1}
I20260928 14:55:47.015555 1233 heartbeater.cc:347] Connected to a master server at 127.20.116.190:43699
I20260928 14:55:47.015681 1233 heartbeater.cc:464] Registering TS with master...
I20260928 14:55:47.015945 1233 heartbeater.cc:511] Master 127.20.116.190:43699 requested a full tablet report, sending...
I20260928 14:55:47.016363 743 ts_manager.cc:194] Registered new tserver with Master: 595b4cbd51a74962a8ec4088e07fd2bb (127.20.116.131:37953)
I20260928 14:55:47.016906 743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:37871
I20260928 14:55:47.023164 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:55:47.023964 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 826
I20260928 14:55:47.032226 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 966
I20260928 14:55:47.037853 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 1103
I20260928 14:55:47.043555 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 712
I20260928 14:55:47.048595 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:43699
--webserver_interface=127.20.116.190
--webserver_port=40445
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:43699
--ranger_config_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.116.148:50385/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.116.148:50385/kms with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:55:47.130609 1236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:47.130825 1236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:47.130848 1236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:47.132010 1236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:55:47.132054 1236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:47.132068 1236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:47.132107 1236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:55:47.132138 1236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:55:47.133695 1236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.116.148:50385/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:43699
--ranger_config_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:43699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=40445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:47.133946 1236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:47.134172 1236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:47.139024 1242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:47.139038 1241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:47.139019 1244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:47.139114 1236 server_base.cc:1034] running on GCE node
I20260928 14:55:47.139496 1236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:47.139796 1236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:47.140966 1236 hybrid_clock.cc:648] HybridClock initialized: now 1790607347140950 us; error 42 us; skew 500 ppm
Sep 28 14:55:47 dist-test-slave-pxl6 krb5kdc[32734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607347, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:47.143507 1236 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:55:47.143972 1236 webserver.cc:460] Webserver started at http://127.20.116.190:40445/ using document root <none> and password file <none>
I20260928 14:55:47.144207 1236 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:47.144279 1236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:55:47 dist-test-slave-pxl6 krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607347, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for HTTP/127.20.116.148@KRBTEST.COM
I20260928 14:55:47.221467 1236 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260928 14:55:47.222611 1252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:47.222872 1236 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:55:47.222982 1236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "902fed5c3300416dbcc2fc1f3aa94320"
format_stamp: "Formatted at 2026-09-28 14:55:45 on dist-test-slave-pxl6"
server_key: "ea72f51dedbeca901d6cc17c4d465ade"
server_key_iv: "3ea5a7b4d7543fd426dde7866e0e7637"
server_key_version: "kuduclusterkey@0"
I20260928 14:55:47.223510 1236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:47.244628 1236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:47.247315 1236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:47.247498 1236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:47.252697 1236 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:43699
I20260928 14:55:47.252765 1304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:43699 every 8 connection(s)
I20260928 14:55:47.253168 1236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:55:47.255155 1305 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:55:47.255270 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 1236
I20260928 14:55:47.255573 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.116.148:50385/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:40603
--local_ip_for_outbound_sockets=127.20.116.129
--tserver_master_addrs=127.20.116.190:43699
--webserver_port=35519
--webserver_interface=127.20.116.129
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.116.148:50385/kms with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:55:47.256273 1305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320: Bootstrap starting.
I20260928 14:55:47.267604 1305 log.cc:824] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320: Log is configured to *not* fsync() on all Append() calls
I20260928 14:55:47.269847 1305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:55:47.270093 1305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320: Bootstrap complete.
I20260928 14:55:47.272526 1305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902fed5c3300416dbcc2fc1f3aa94320" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43699 } }
I20260928 14:55:47.272776 1305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 902fed5c3300416dbcc2fc1f3aa94320, State: Initialized, Role: FOLLOWER
I20260928 14:55:47.272953 1305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902fed5c3300416dbcc2fc1f3aa94320" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43699 } }
I20260928 14:55:47.273026 1305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:55:47.273083 1305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:55:47.273128 1305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:55:47.274140 1305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902fed5c3300416dbcc2fc1f3aa94320" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43699 } }
I20260928 14:55:47.274272 1305 leader_election.cc:302] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 902fed5c3300416dbcc2fc1f3aa94320; no voters:
I20260928 14:55:47.274449 1305 leader_election.cc:288] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:55:47.274559 1309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:55:47.274730 1305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:55:47.274775 1309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [term 2 LEADER]: Becoming Leader. State: Replica: 902fed5c3300416dbcc2fc1f3aa94320, State: Running, Role: LEADER
I20260928 14:55:47.274967 1309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902fed5c3300416dbcc2fc1f3aa94320" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43699 } }
I20260928 14:55:47.275251 1309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 902fed5c3300416dbcc2fc1f3aa94320. Latest consensus state: current_term: 2 leader_uuid: "902fed5c3300416dbcc2fc1f3aa94320" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902fed5c3300416dbcc2fc1f3aa94320" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43699 } } }
I20260928 14:55:47.275357 1309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [sys.catalog]: This master's current role is: LEADER
I20260928 14:55:47.275496 1310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "902fed5c3300416dbcc2fc1f3aa94320" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902fed5c3300416dbcc2fc1f3aa94320" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43699 } } }
I20260928 14:55:47.275550 1310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320 [sys.catalog]: This master's current role is: LEADER
I20260928 14:55:47.275610 1317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:55:47.276196 1317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:55:47.276360 1317 catalog_manager.cc:1259] Loaded cluster ID: 5278728b318d4b8292cd12137f1c1167
I20260928 14:55:47.276402 1317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:55:47.276556 1305 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 14:55:47.280253 1317 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:55:47.280871 1317 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320: Loaded TSK: 0
I20260928 14:55:47.281148 1317 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:55:47.348704 1307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:47.348928 1307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:47.348970 1307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:47.350159 1307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:47.350226 1307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:47.350271 1307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:55:47.352242 1307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.116.148:50385/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:40603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=35519
--tserver_master_addrs=127.20.116.190:43699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:47.352511 1307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:47.352785 1307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:47.359115 1343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:47.359115 1345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:47.359690 1342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:47.359949 1307 server_base.cc:1034] running on GCE node
I20260928 14:55:47.360241 1307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:47.360499 1307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:47.361680 1307 hybrid_clock.cc:648] HybridClock initialized: now 1790607347361626 us; error 68 us; skew 500 ppm
Sep 28 14:55:47 dist-test-slave-pxl6 krb5kdc[32734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607347, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:47.364454 1307 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:55:47.364912 1307 webserver.cc:460] Webserver started at http://127.20.116.129:35519/ using document root <none> and password file <none>
I20260928 14:55:47.365206 1307 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:47.365298 1307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:55:47 dist-test-slave-pxl6 krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607347, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for HTTP/127.20.116.148@KRBTEST.COM
I20260928 14:55:47.392796 1307 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.001s sys 0.000s
I20260928 14:55:47.393875 1354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:47.394124 1307 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:55:47.394224 1307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "7647f0c709974cfa854d509ee77ae2ff"
format_stamp: "Formatted at 2026-09-28 14:55:46 on dist-test-slave-pxl6"
server_key: "12733cac10f2386ffe8181825a724d7d"
server_key_iv: "79e4ae69a4512cdb45fe50867ccb398a"
server_key_version: "kuduclusterkey@0"
I20260928 14:55:47.394698 1307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:47.410115 1307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:47.412788 1307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:47.412968 1307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:47.413251 1307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:55:47.413678 1307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:55:47.413727 1307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:47.413766 1307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:55:47.413791 1307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:47.421984 1307 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:40603
I20260928 14:55:47.422457 1307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:55:47.423175 1468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:40603 every 8 connection(s)
I20260928 14:55:47.423261 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 1307
I20260928 14:55:47.423563 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.116.148:50385/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:35027
--local_ip_for_outbound_sockets=127.20.116.130
--tserver_master_addrs=127.20.116.190:43699
--webserver_port=40437
--webserver_interface=127.20.116.130
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.116.148:50385/kms with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:55:47 dist-test-slave-pxl6 krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607347, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:55:47.450096 1471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:47.424237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43699 (local address 127.20.116.129:38485)
0928 14:55:47.424654 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:47.424695 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:47.425435 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:47.427488 (+ 2053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:47.427493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:47.427758 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:47.428041 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:47.428048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:47.429830 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:47.429831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:47.431108 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:47.431111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:47.431990 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:47.431992 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:47.432133 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:47.432785 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:47.432795 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:47.434466 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:47.448030 (+ 13564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:47.448044 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:47.448047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:47.448419 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:47.448792 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:47.448793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:47.448793 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:47.448941 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:47.449401 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:47.449406 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:47.449571 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 14:55:47.449675 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 14:55:47.449849 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":94,"threads_started":1}
I20260928 14:55:47.450933 1473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:47.425425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:38485 (local address 127.20.116.190:43699)
0928 14:55:47.426986 (+ 1561us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:47.426996 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:47.427041 (+ 45us) server_negotiation.cc:407] Connection header received
0928 14:55:47.427204 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:47.427212 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:47.427273 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:47.428175 (+ 902us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:47.428276 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:47.429710 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:47.431224 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:47.431857 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:47.433496 (+ 1639us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:47.434652 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:47.434657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:47.434658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:47.434670 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:47.447878 (+ 13208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:47.448541 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:47.448544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:47.448545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:47.448698 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:47.449017 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:47.449017 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:47.449018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:47.449317 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:47.450446 (+ 1129us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:47.450525 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 14:55:47.450678 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1500,"thread_start_us":83,"threads_started":1}
I20260928 14:55:47.451822 1469 heartbeater.cc:347] Connected to a master server at 127.20.116.190:43699
I20260928 14:55:47.451936 1469 heartbeater.cc:464] Registering TS with master...
I20260928 14:55:47.452152 1469 heartbeater.cc:511] Master 127.20.116.190:43699 requested a full tablet report, sending...
W20260928 14:55:47.453066 1469 heartbeater.cc:644] Failed to heartbeat to 127.20.116.190:43699 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 14:55:47.453140 1469 heartbeater.cc:464] Registering TS with master...
I20260928 14:55:47.453207 1469 heartbeater.cc:511] Master 127.20.116.190:43699 requested a full tablet report, sending...
I20260928 14:55:47.453526 1469 heartbeater.cc:464] Registering TS with master...
I20260928 14:55:47.453613 1469 heartbeater.cc:511] Master 127.20.116.190:43699 requested a full tablet report, sending...
W20260928 14:55:47.453938 1469 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:55:47.559981 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:47.560297 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:47.560348 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:47.562207 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:47.562282 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:47.562328 1472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:55:47.564628 1472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.116.148:50385/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:35027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=40437
--tserver_master_addrs=127.20.116.190:43699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:47.564911 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:47.565176 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:47.571435 1479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:47.571444 1478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:47.571916 1481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:47.572860 1472 server_base.cc:1034] running on GCE node
I20260928 14:55:47.573103 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:47.573357 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:47.574512 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790607347574490 us; error 33 us; skew 500 ppm
Sep 28 14:55:47 dist-test-slave-pxl6 krb5kdc[32734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607347, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:47.577862 1472 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:55:47.578403 1472 webserver.cc:460] Webserver started at http://127.20.116.130:40437/ using document root <none> and password file <none>
I20260928 14:55:47.578716 1472 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:47.578810 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:55:47 dist-test-slave-pxl6 krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607347, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for HTTP/127.20.116.148@KRBTEST.COM
I20260928 14:55:47.603309 1472 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:55:47.604005 1489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:47.604151 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:55:47.604233 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "212b9b577ee04d11a86a9f319911ba39"
format_stamp: "Formatted at 2026-09-28 14:55:46 on dist-test-slave-pxl6"
server_key: "af205b1c13755d1a6d0c72dfe78871b6"
server_key_iv: "f3cde576d7ce5d54c95f5929733808ae"
server_key_version: "kuduclusterkey@0"
I20260928 14:55:47.605317 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:47.629691 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:47.632176 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:47.632336 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:47.632550 1472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:55:47.632907 1472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:55:47.632951 1472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:47.632989 1472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:55:47.633008 1472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:47.640218 1472 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:35027
I20260928 14:55:47.640779 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:55:47.642793 1603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:35027 every 8 connection(s)
I20260928 14:55:47.646929 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 1472
I20260928 14:55:47.647256 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.116.148:50385/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:37953
--local_ip_for_outbound_sockets=127.20.116.131
--tserver_master_addrs=127.20.116.190:43699
--webserver_port=39547
--webserver_interface=127.20.116.131
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.116.148:50385/kms with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:55:47 dist-test-slave-pxl6 krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607347, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:55:47.654747 1473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:47.643890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:33131 (local address 127.20.116.190:43699)
0928 14:55:47.643939 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:47.643942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:47.643995 (+ 53us) server_negotiation.cc:407] Connection header received
0928 14:55:47.644742 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:47.644744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:47.644763 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:47.644795 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:47.645676 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:47.646402 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:47.647927 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:47.648231 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:47.648274 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:47.650993 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:47.650995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:47.650996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:47.651004 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:47.652789 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:47.653412 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:47.653414 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:47.653414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:47.653487 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:47.653813 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:47.653814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:47.653814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:47.653972 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:47.654043 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:47.654498 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 14:55:47.654644 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 14:55:47.655278 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:47.643543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43699 (local address 127.20.116.130:33131)
0928 14:55:47.643843 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:47.643899 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:47.644541 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:47.645007 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:47.645012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:47.645257 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:47.645549 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:47.645556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:47.646522 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:47.646523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:47.647821 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:47.647824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:47.648386 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:47.648387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:47.648534 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:47.649178 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:47.649189 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:47.650875 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:47.652913 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:47.652922 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:47.652926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:47.653302 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:47.653566 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:47.653567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:47.653567 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:47.653728 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:47.654080 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:47.654087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:47.654287 (+ 200us) client_negotiation.cc:769] Sending connection context
0928 14:55:47.654853 (+ 566us) client_negotiation.cc:240] Negotiation successful
0928 14:55:47.655030 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":106,"threads_started":1}
I20260928 14:55:47.655850 1604 heartbeater.cc:347] Connected to a master server at 127.20.116.190:43699
I20260928 14:55:47.655961 1604 heartbeater.cc:464] Registering TS with master...
I20260928 14:55:47.656196 1604 heartbeater.cc:511] Master 127.20.116.190:43699 requested a full tablet report, sending...
W20260928 14:55:47.656992 1604 heartbeater.cc:644] Failed to heartbeat to 127.20.116.190:43699 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 14:55:47.657105 1604 heartbeater.cc:464] Registering TS with master...
I20260928 14:55:47.657196 1604 heartbeater.cc:511] Master 127.20.116.190:43699 requested a full tablet report, sending...
I20260928 14:55:47.657632 1604 heartbeater.cc:464] Registering TS with master...
I20260928 14:55:47.657714 1604 heartbeater.cc:511] Master 127.20.116.190:43699 requested a full tablet report, sending...
W20260928 14:55:47.658069 1604 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:55:47.769766 1607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:47.770022 1607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:47.770061 1607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:47.771605 1607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:47.771661 1607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:47.771713 1607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:55:47.773736 1607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.116.148:50385/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:37953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=39547
--tserver_master_addrs=127.20.116.190:43699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:47.774047 1607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:47.774350 1607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:47.781682 1615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:47.781773 1612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:47.781677 1613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:47.789765 1607 server_base.cc:1034] running on GCE node
I20260928 14:55:47.790122 1607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:47.790467 1607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:47.797838 1607 hybrid_clock.cc:648] HybridClock initialized: now 1790607347797775 us; error 109 us; skew 500 ppm
Sep 28 14:55:47 dist-test-slave-pxl6 krb5kdc[32734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607347, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:47.801419 1607 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:55:47.802021 1607 webserver.cc:460] Webserver started at http://127.20.116.131:39547/ using document root <none> and password file <none>
I20260928 14:55:47.802327 1607 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:47.802392 1607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:55:47 dist-test-slave-pxl6 krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607347, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for HTTP/127.20.116.148@KRBTEST.COM
I20260928 14:55:47.832031 1607 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260928 14:55:47.832901 1623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:47.833117 1607 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:55:47.833210 1607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "595b4cbd51a74962a8ec4088e07fd2bb"
format_stamp: "Formatted at 2026-09-28 14:55:46 on dist-test-slave-pxl6"
server_key: "cb49a14ab60acd3c8f21f122cdc77adf"
server_key_iv: "9e0e33349c332d5fb2cdb102408e2603"
server_key_version: "kuduclusterkey@0"
I20260928 14:55:47.834394 1607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:47.848426 1607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:47.851035 1607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:47.851261 1607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:47.851531 1607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:55:47.851971 1607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:55:47.852056 1607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:47.852145 1607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:55:47.852206 1607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:47.859061 1607 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:37953
I20260928 14:55:47.859617 1607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:55:47.861260 1738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:37953 every 8 connection(s)
I20260928 14:55:47.863727 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 1607
Sep 28 14:55:47 dist-test-slave-pxl6 krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607347, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:55:47.873387 1473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:47.861978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:60227 (local address 127.20.116.190:43699)
0928 14:55:47.862030 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:47.862033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:47.862518 (+ 485us) server_negotiation.cc:407] Connection header received
0928 14:55:47.863269 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:47.863270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:47.863285 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:47.863320 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:47.864252 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:47.864962 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:47.866413 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:47.866727 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:47.866775 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:47.869993 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:47.869995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:47.869995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:47.870005 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:47.871804 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:47.872299 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:47.872301 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:47.872301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:47.872361 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:47.872630 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:47.872631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:47.872631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:47.872761 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:47.872802 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:47.873191 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 14:55:47.873285 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:55:47.873778 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:47.861981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43699 (local address 127.20.116.131:60227)
0928 14:55:47.862363 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:47.862418 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:47.863135 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:47.863513 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:47.863518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:47.863776 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:47.864133 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:47.864140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:47.865086 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:47.865087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:47.866306 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:47.866309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:47.866897 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:47.866898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:47.867074 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:47.867740 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:47.867753 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:47.869859 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:47.871901 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:47.871908 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:47.871911 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:47.872209 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:47.872430 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:47.872430 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:47.872431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:47.872559 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:47.872848 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:47.872853 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:47.873019 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 14:55:47.873131 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 14:55:47.873296 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":83,"threads_started":1}
I20260928 14:55:47.874233 1739 heartbeater.cc:347] Connected to a master server at 127.20.116.190:43699
I20260928 14:55:47.874336 1739 heartbeater.cc:464] Registering TS with master...
I20260928 14:55:47.874562 1739 heartbeater.cc:511] Master 127.20.116.190:43699 requested a full tablet report, sending...
W20260928 14:55:47.875154 1739 heartbeater.cc:644] Failed to heartbeat to 127.20.116.190:43699 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 14:55:47.875231 1739 heartbeater.cc:464] Registering TS with master...
I20260928 14:55:47.875288 1739 heartbeater.cc:511] Master 127.20.116.190:43699 requested a full tablet report, sending...
I20260928 14:55:47.875603 1739 heartbeater.cc:464] Registering TS with master...
I20260928 14:55:47.875689 1739 heartbeater.cc:511] Master 127.20.116.190:43699 requested a full tablet report, sending...
W20260928 14:55:47.876056 1739 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:55:47.880947 1473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:47.864121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44148 (local address 127.20.116.190:43699)
0928 14:55:47.873474 (+ 9353us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:47.873476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:47.873485 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:55:47.873504 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:47.873504 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:47.873516 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:47.873596 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:47.874409 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:47.875191 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:47.876377 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:47.876668 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:47.876739 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:47.878146 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:47.878147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:47.878148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:47.878155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:47.879864 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:47.880248 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:47.880249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:47.880250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:47.880310 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:47.880495 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:47.880496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:47.880496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:47.880631 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:47.880792 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:47.880826 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 14:55:47.880884 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9337,"thread_start_us":61,"threads_started":1}
Sep 28 14:55:48 dist-test-slave-pxl6 krb5kdc[32734](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607348, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:48.454159 1469 heartbeater.cc:464] Registering TS with master...
I20260928 14:55:48.454312 1469 heartbeater.cc:511] Master 127.20.116.190:43699 requested a full tablet report, sending...
I20260928 14:55:48.454757 1268 ts_manager.cc:194] Registered new tserver with Master: 7647f0c709974cfa854d509ee77ae2ff (127.20.116.129:40603)
I20260928 14:55:48.455790 1268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:38485
I20260928 14:55:48.658303 1604 heartbeater.cc:464] Registering TS with master...
I20260928 14:55:48.658490 1604 heartbeater.cc:511] Master 127.20.116.190:43699 requested a full tablet report, sending...
I20260928 14:55:48.658900 1268 ts_manager.cc:194] Registered new tserver with Master: 212b9b577ee04d11a86a9f319911ba39 (127.20.116.130:35027)
I20260928 14:55:48.659468 1268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:33131
Sep 28 14:55:48 dist-test-slave-pxl6 krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607348, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for HTTP/127.20.116.148@KRBTEST.COM
I20260928 14:55:48.876240 1739 heartbeater.cc:464] Registering TS with master...
I20260928 14:55:48.876420 1739 heartbeater.cc:511] Master 127.20.116.190:43699 requested a full tablet report, sending...
I20260928 14:55:48.876812 1268 ts_manager.cc:194] Registered new tserver with Master: 595b4cbd51a74962a8ec4088e07fd2bb (127.20.116.131:37953)
I20260928 14:55:48.877365 1268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:60227
I20260928 14:55:48.879307 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:55:48.889472 1760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:48.880303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.20.116.190:43699)
0928 14:55:48.880581 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:48.880584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:48.880761 (+ 177us) server_negotiation.cc:407] Connection header received
0928 14:55:48.880786 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:48.880788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:48.880800 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:48.880857 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:48.881650 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.882568 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.883650 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.883969 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.884027 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:48.885621 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:48.885623 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:48.885623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:48.885631 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:48.887928 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:48.888342 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:48.888343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:48.888343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:48.888403 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:48.888590 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:48.888591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:48.888591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:48.888722 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:48.888820 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:48.889277 (+ 457us) server_negotiation.cc:299] Negotiation successful
0928 14:55:48.889359 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":68,"threads_started":1}
I20260928 14:55:48.891850 1268 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:55:48.894675 1268 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:55:48.904043 1766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:48.897644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:37953 (local address 127.0.0.1:41276)
0928 14:55:48.898793 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:48.898836 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:48.898895 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:48.899251 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:48.899254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:48.899257 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:55:48.899323 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:48.899325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.900902 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.900903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:48.902249 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:48.902252 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.903699 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.903700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:48.903803 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:48.903819 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 14:55:48.903897 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:55:48.903951 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"thread_start_us":74,"threads_started":1}
I20260928 14:55:48.904644 1769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:48.898126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41276 (local address 127.20.116.131:37953)
0928 14:55:48.899019 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:48.899025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:48.899034 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:55:48.899068 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:48.899070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:48.899085 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:48.899151 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:55:48.899391 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.900823 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.902351 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.903479 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.903556 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:48.903658 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:48.904518 (+ 860us) server_negotiation.cc:299] Negotiation successful
0928 14:55:48.904563 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":862,"thread_start_us":69,"threads_started":1}
I20260928 14:55:48.905406 1672 tablet_service.cc:1467] Processing CreateTablet for tablet 6b5801ca5d24491193ce95a473dc06da (DEFAULT_TABLE table=test-table [id=505b109c1e424a9ba4266c6c6f252bc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:55:48.905825 1672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b5801ca5d24491193ce95a473dc06da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:55:48.915315 1766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:48.898411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:40603 (local address 127.0.0.1:56632)
0928 14:55:48.904398 (+ 5987us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:48.904409 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:48.904443 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:48.906992 (+ 2549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:48.906993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:48.906995 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:55:48.907030 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:48.907031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.910185 (+ 3154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.910186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:48.911371 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:48.911374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.915040 (+ 3666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.915041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:48.915105 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:48.915110 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:55:48.915140 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 14:55:48.915230 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5978,"thread_start_us":67,"threads_started":1}
I20260928 14:55:48.916015 1771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:48.898497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56632 (local address 127.20.116.129:40603)
0928 14:55:48.906740 (+ 8243us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:48.906745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:48.906758 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:55:48.906795 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:48.906797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:48.906814 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:48.906906 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:55:48.907107 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.908148 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.911483 (+ 3335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.912322 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.915767 (+ 3445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:48.915857 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:48.915888 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 14:55:48.915937 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8202,"thread_start_us":69,"threads_started":1}
I20260928 14:55:48.916741 1770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:48.897871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:35027 (local address 127.0.0.1:58694)
0928 14:55:48.900625 (+ 2754us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:48.900635 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:48.900680 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:48.908497 (+ 7817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:48.908499 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:48.908501 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:55:48.908545 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:48.908547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.912435 (+ 3888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.912436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:48.913975 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:48.913978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.916518 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.916519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:48.916595 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:48.916599 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:55:48.916632 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 14:55:48.916673 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2737,"thread_start_us":87,"threads_started":1}
I20260928 14:55:48.916939 1402 tablet_service.cc:1467] Processing CreateTablet for tablet 6b5801ca5d24491193ce95a473dc06da (DEFAULT_TABLE table=test-table [id=505b109c1e424a9ba4266c6c6f252bc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:55:48.917361 1402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b5801ca5d24491193ce95a473dc06da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:55:48.917384 1768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:48.897976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58694 (local address 127.20.116.130:35027)
0928 14:55:48.908316 (+ 10340us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:48.908320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:48.908330 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:55:48.908364 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:48.908366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:48.908383 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:48.908616 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:55:48.908702 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.910055 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.914087 (+ 4032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.914931 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.917074 (+ 2143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:48.917218 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:48.917247 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 14:55:48.917304 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10301,"thread_start_us":52,"threads_started":1}
I20260928 14:55:48.918813 1772 tablet_bootstrap.cc:492] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb: Bootstrap starting.
I20260928 14:55:48.919564 1772 tablet_bootstrap.cc:654] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb: Neither blocks nor log segments found. Creating new log.
I20260928 14:55:48.919930 1772 log.cc:824] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb: Log is configured to *not* fsync() on all Append() calls
I20260928 14:55:48.921577 1537 tablet_service.cc:1467] Processing CreateTablet for tablet 6b5801ca5d24491193ce95a473dc06da (DEFAULT_TABLE table=test-table [id=505b109c1e424a9ba4266c6c6f252bc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:55:48.921945 1537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b5801ca5d24491193ce95a473dc06da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:55:48.930742 1774 tablet_bootstrap.cc:492] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff: Bootstrap starting.
I20260928 14:55:48.931245 1772 tablet_bootstrap.cc:492] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb: No bootstrap required, opened a new log
I20260928 14:55:48.931329 1772 ts_tablet_manager.cc:1397] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260928 14:55:48.931458 1774 tablet_bootstrap.cc:654] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff: Neither blocks nor log segments found. Creating new log.
I20260928 14:55:48.931782 1774 log.cc:824] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff: Log is configured to *not* fsync() on all Append() calls
I20260928 14:55:48.932116 1775 tablet_bootstrap.cc:492] T 6b5801ca5d24491193ce95a473dc06da P 212b9b577ee04d11a86a9f319911ba39: Bootstrap starting.
I20260928 14:55:48.932744 1775 tablet_bootstrap.cc:654] T 6b5801ca5d24491193ce95a473dc06da P 212b9b577ee04d11a86a9f319911ba39: Neither blocks nor log segments found. Creating new log.
I20260928 14:55:48.933030 1775 log.cc:824] T 6b5801ca5d24491193ce95a473dc06da P 212b9b577ee04d11a86a9f319911ba39: Log is configured to *not* fsync() on all Append() calls
I20260928 14:55:48.933806 1772 raft_consensus.cc:348] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595b4cbd51a74962a8ec4088e07fd2bb" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37953 } } peers { permanent_uuid: "7647f0c709974cfa854d509ee77ae2ff" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40603 } } peers { permanent_uuid: "212b9b577ee04d11a86a9f319911ba39" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35027 } }
I20260928 14:55:48.933991 1772 raft_consensus.cc:374] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:55:48.934029 1772 raft_consensus.cc:729] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 595b4cbd51a74962a8ec4088e07fd2bb, State: Initialized, Role: FOLLOWER
I20260928 14:55:48.934185 1772 consensus_queue.cc:260] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595b4cbd51a74962a8ec4088e07fd2bb" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37953 } } peers { permanent_uuid: "7647f0c709974cfa854d509ee77ae2ff" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40603 } } peers { permanent_uuid: "212b9b577ee04d11a86a9f319911ba39" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35027 } }
I20260928 14:55:48.934473 1772 ts_tablet_manager.cc:1428] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:55:48.934677 1739 heartbeater.cc:503] Master 127.20.116.190:43699 was elected leader, sending a full tablet report...
I20260928 14:55:48.939314 1774 tablet_bootstrap.cc:492] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff: No bootstrap required, opened a new log
I20260928 14:55:48.939414 1774 ts_tablet_manager.cc:1397] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260928 14:55:48.941427 1774 raft_consensus.cc:348] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595b4cbd51a74962a8ec4088e07fd2bb" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37953 } } peers { permanent_uuid: "7647f0c709974cfa854d509ee77ae2ff" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40603 } } peers { permanent_uuid: "212b9b577ee04d11a86a9f319911ba39" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35027 } }
I20260928 14:55:48.941599 1774 raft_consensus.cc:374] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:55:48.941642 1774 raft_consensus.cc:729] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7647f0c709974cfa854d509ee77ae2ff, State: Initialized, Role: FOLLOWER
I20260928 14:55:48.941802 1774 consensus_queue.cc:260] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595b4cbd51a74962a8ec4088e07fd2bb" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37953 } } peers { permanent_uuid: "7647f0c709974cfa854d509ee77ae2ff" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40603 } } peers { permanent_uuid: "212b9b577ee04d11a86a9f319911ba39" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35027 } }
I20260928 14:55:48.942111 1774 ts_tablet_manager.cc:1428] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:55:48.942144 1469 heartbeater.cc:503] Master 127.20.116.190:43699 was elected leader, sending a full tablet report...
I20260928 14:55:48.943204 1775 tablet_bootstrap.cc:492] T 6b5801ca5d24491193ce95a473dc06da P 212b9b577ee04d11a86a9f319911ba39: No bootstrap required, opened a new log
I20260928 14:55:48.943313 1775 ts_tablet_manager.cc:1397] T 6b5801ca5d24491193ce95a473dc06da P 212b9b577ee04d11a86a9f319911ba39: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260928 14:55:48.944990 1775 raft_consensus.cc:348] T 6b5801ca5d24491193ce95a473dc06da P 212b9b577ee04d11a86a9f319911ba39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595b4cbd51a74962a8ec4088e07fd2bb" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37953 } } peers { permanent_uuid: "7647f0c709974cfa854d509ee77ae2ff" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40603 } } peers { permanent_uuid: "212b9b577ee04d11a86a9f319911ba39" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35027 } }
I20260928 14:55:48.945152 1775 raft_consensus.cc:374] T 6b5801ca5d24491193ce95a473dc06da P 212b9b577ee04d11a86a9f319911ba39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:55:48.945199 1775 raft_consensus.cc:729] T 6b5801ca5d24491193ce95a473dc06da P 212b9b577ee04d11a86a9f319911ba39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b9b577ee04d11a86a9f319911ba39, State: Initialized, Role: FOLLOWER
I20260928 14:55:48.945320 1775 consensus_queue.cc:260] T 6b5801ca5d24491193ce95a473dc06da P 212b9b577ee04d11a86a9f319911ba39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595b4cbd51a74962a8ec4088e07fd2bb" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37953 } } peers { permanent_uuid: "7647f0c709974cfa854d509ee77ae2ff" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40603 } } peers { permanent_uuid: "212b9b577ee04d11a86a9f319911ba39" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35027 } }
I20260928 14:55:48.945623 1775 ts_tablet_manager.cc:1428] T 6b5801ca5d24491193ce95a473dc06da P 212b9b577ee04d11a86a9f319911ba39: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:55:48.945657 1604 heartbeater.cc:503] Master 127.20.116.190:43699 was elected leader, sending a full tablet report...
I20260928 14:55:48.957683 1779 raft_consensus.cc:482] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:55:48.957845 1779 raft_consensus.cc:504] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595b4cbd51a74962a8ec4088e07fd2bb" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37953 } } peers { permanent_uuid: "7647f0c709974cfa854d509ee77ae2ff" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40603 } } peers { permanent_uuid: "212b9b577ee04d11a86a9f319911ba39" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35027 } }
I20260928 14:55:48.958278 1779 leader_election.cc:288] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 595b4cbd51a74962a8ec4088e07fd2bb (127.20.116.131:37953), 212b9b577ee04d11a86a9f319911ba39 (127.20.116.130:35027)
I20260928 14:55:48.962867 1768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:48.958714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:33153 (local address 127.20.116.130:35027)
0928 14:55:48.958768 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:48.958770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:48.958979 (+ 209us) server_negotiation.cc:407] Connection header received
0928 14:55:48.959084 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:48.959085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:48.959094 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:48.959127 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:55:48.959384 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.960165 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.961808 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.962383 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.962433 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:48.962441 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:48.962734 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 14:55:48.962789 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:55:48.962867 1769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:48.958674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:43317 (local address 127.20.116.131:37953)
0928 14:55:48.958723 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:48.958725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:48.958969 (+ 244us) server_negotiation.cc:407] Connection header received
0928 14:55:48.959084 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:48.959085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:48.959094 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:48.959127 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:55:48.959386 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.960150 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.961800 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.962383 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.962433 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:48.962441 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:48.962734 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 14:55:48.962781 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:55:48.963128 1781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:48.958424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:37953 (local address 127.20.116.129:43317)
0928 14:55:48.958893 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:48.958907 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:48.959001 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:48.959201 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:48.959202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:48.959206 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:55:48.959307 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:48.959309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.960266 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.960267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:48.961685 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:48.961688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.962563 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.962564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:48.962641 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:48.962647 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:55:48.962957 (+ 310us) client_negotiation.cc:240] Negotiation successful
0928 14:55:48.963010 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":106,"thread_start_us":438,"threads_started":1}
I20260928 14:55:48.963502 1782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:48.958445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:35027 (local address 127.20.116.129:33153)
0928 14:55:48.958922 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:48.958932 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:48.959001 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:48.959201 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:48.959202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:48.959204 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:55:48.959300 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:48.959302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.960264 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.960265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:48.961684 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:48.961688 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.962557 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.962558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:48.962641 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:48.962648 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 14:55:48.963379 (+ 731us) client_negotiation.cc:240] Negotiation successful
0928 14:55:48.963426 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":69,"thread_start_us":139,"threads_started":1}
I20260928 14:55:48.963513 1692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5801ca5d24491193ce95a473dc06da" candidate_uuid: "7647f0c709974cfa854d509ee77ae2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "595b4cbd51a74962a8ec4088e07fd2bb" is_pre_election: true
I20260928 14:55:48.963716 1692 raft_consensus.cc:2454] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7647f0c709974cfa854d509ee77ae2ff in term 0.
I20260928 14:55:48.963841 1555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5801ca5d24491193ce95a473dc06da" candidate_uuid: "7647f0c709974cfa854d509ee77ae2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b9b577ee04d11a86a9f319911ba39" is_pre_election: true
I20260928 14:55:48.963960 1356 leader_election.cc:302] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 595b4cbd51a74962a8ec4088e07fd2bb, 7647f0c709974cfa854d509ee77ae2ff; no voters:
I20260928 14:55:48.964007 1555 raft_consensus.cc:2454] T 6b5801ca5d24491193ce95a473dc06da P 212b9b577ee04d11a86a9f319911ba39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7647f0c709974cfa854d509ee77ae2ff in term 0.
I20260928 14:55:48.964107 1779 raft_consensus.cc:2781] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:55:48.964164 1779 raft_consensus.cc:482] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:55:48.964195 1779 raft_consensus.cc:3037] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:55:48.964896 1779 raft_consensus.cc:504] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595b4cbd51a74962a8ec4088e07fd2bb" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37953 } } peers { permanent_uuid: "7647f0c709974cfa854d509ee77ae2ff" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40603 } } peers { permanent_uuid: "212b9b577ee04d11a86a9f319911ba39" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35027 } }
I20260928 14:55:48.965049 1779 leader_election.cc:288] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff [CANDIDATE]: Term 1 election: Requested vote from peers 595b4cbd51a74962a8ec4088e07fd2bb (127.20.116.131:37953), 212b9b577ee04d11a86a9f319911ba39 (127.20.116.130:35027)
I20260928 14:55:48.965364 1555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5801ca5d24491193ce95a473dc06da" candidate_uuid: "7647f0c709974cfa854d509ee77ae2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b9b577ee04d11a86a9f319911ba39"
I20260928 14:55:48.965399 1692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5801ca5d24491193ce95a473dc06da" candidate_uuid: "7647f0c709974cfa854d509ee77ae2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "595b4cbd51a74962a8ec4088e07fd2bb"
I20260928 14:55:48.965448 1555 raft_consensus.cc:3037] T 6b5801ca5d24491193ce95a473dc06da P 212b9b577ee04d11a86a9f319911ba39 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:55:48.965462 1692 raft_consensus.cc:3037] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:55:48.966347 1555 raft_consensus.cc:2454] T 6b5801ca5d24491193ce95a473dc06da P 212b9b577ee04d11a86a9f319911ba39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7647f0c709974cfa854d509ee77ae2ff in term 1.
I20260928 14:55:48.966347 1692 raft_consensus.cc:2454] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7647f0c709974cfa854d509ee77ae2ff in term 1.
I20260928 14:55:48.966519 1356 leader_election.cc:302] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 595b4cbd51a74962a8ec4088e07fd2bb, 7647f0c709974cfa854d509ee77ae2ff; no voters:
I20260928 14:55:48.966719 1779 raft_consensus.cc:2781] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:55:48.966923 1779 raft_consensus.cc:686] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff [term 1 LEADER]: Becoming Leader. State: Replica: 7647f0c709974cfa854d509ee77ae2ff, State: Running, Role: LEADER
I20260928 14:55:48.967027 1779 consensus_queue.cc:237] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595b4cbd51a74962a8ec4088e07fd2bb" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37953 } } peers { permanent_uuid: "7647f0c709974cfa854d509ee77ae2ff" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40603 } } peers { permanent_uuid: "212b9b577ee04d11a86a9f319911ba39" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35027 } }
I20260928 14:55:48.967854 1269 catalog_manager.cc:5495] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 7647f0c709974cfa854d509ee77ae2ff (127.20.116.129). New cstate: current_term: 1 leader_uuid: "7647f0c709974cfa854d509ee77ae2ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595b4cbd51a74962a8ec4088e07fd2bb" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7647f0c709974cfa854d509ee77ae2ff" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b9b577ee04d11a86a9f319911ba39" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35027 } health_report { overall_health: UNKNOWN } } }
I20260928 14:55:48.978771 1771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:48.975351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.20.116.129:40603)
0928 14:55:48.975392 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:48.975394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:48.975402 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:55:48.975459 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:48.975459 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:48.975479 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:48.975542 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:55:48.975778 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.976488 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.977579 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:48.977869 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:48.977938 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:48.978144 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:55:48.978295 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:55:48.978373 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:48.978655 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 14:55:48.978678 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:55:48.981302 1555 raft_consensus.cc:1261] T 6b5801ca5d24491193ce95a473dc06da P 212b9b577ee04d11a86a9f319911ba39 [term 1 FOLLOWER]: Refusing update from remote peer 7647f0c709974cfa854d509ee77ae2ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:55:48.981771 1779 consensus_queue.cc:1035] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b9b577ee04d11a86a9f319911ba39" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 35027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:55:48.984278 1692 raft_consensus.cc:1261] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb [term 1 FOLLOWER]: Refusing update from remote peer 7647f0c709974cfa854d509ee77ae2ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:55:48.984637 1779 consensus_queue.cc:1035] T 6b5801ca5d24491193ce95a473dc06da P 7647f0c709974cfa854d509ee77ae2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "595b4cbd51a74962a8ec4088e07fd2bb" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:55:48.988776 1786 mvcc.cc:205] Tried to move back new op lower bound from 7334327701424140288 to 7334327701369544704. Current Snapshot: MvccSnapshot[applied={T|T < 7334327701424140288}]
I20260928 14:55:48.989704 1787 mvcc.cc:205] Tried to move back new op lower bound from 7334327701424140288 to 7334327701369544704. Current Snapshot: MvccSnapshot[applied={T|T < 7334327701424140288}]
I20260928 14:55:48.992285 1269 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:55:48.992386 1269 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:55:48.993485 1269 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 902fed5c3300416dbcc2fc1f3aa94320: Sending DeleteTablet for 3 replicas of tablet 6b5801ca5d24491193ce95a473dc06da
I20260928 14:55:48.993867 1672 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b5801ca5d24491193ce95a473dc06da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:55:48 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:41276
I20260928 14:55:48.993933 1537 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b5801ca5d24491193ce95a473dc06da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:55:48 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:58694
I20260928 14:55:48.994392 1402 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b5801ca5d24491193ce95a473dc06da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:55:48 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:56632
I20260928 14:55:48.995034 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 1307
I20260928 14:55:48.998653 1795 tablet_replica.cc:330] T 6b5801ca5d24491193ce95a473dc06da P 212b9b577ee04d11a86a9f319911ba39: stopping tablet replica
I20260928 14:55:48.998781 1795 raft_consensus.cc:2229] T 6b5801ca5d24491193ce95a473dc06da P 212b9b577ee04d11a86a9f319911ba39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:55:48.998903 1795 raft_consensus.cc:2258] T 6b5801ca5d24491193ce95a473dc06da P 212b9b577ee04d11a86a9f319911ba39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:55:49.003475 1795 ts_tablet_manager.cc:1905] T 6b5801ca5d24491193ce95a473dc06da P 212b9b577ee04d11a86a9f319911ba39: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 14:55:49.004969 1626 connection.cc:535] server connection from 127.20.116.129:43317 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:55:49.005193 1794 tablet_replica.cc:330] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb: stopping tablet replica
I20260928 14:55:49.005322 1794 raft_consensus.cc:2229] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:55:49.005424 1794 pending_rounds.cc:62] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb: Trying to abort 2 pending ops.
I20260928 14:55:49.005468 1794 pending_rounds.cc:68] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7334327701424140288 op_type: WRITE_OP write_request { tablet_id: "6b5801ca5d24491193ce95a473dc06da" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\240\202\352\325\006\"8\n\ntest-admin\022*\n 505b109c1e424a9ba4266c6c6f252bc8\020\001\030\001 \001(\001" signature: "<redacted>""\222\036q\212|\241\024\033Ide\203\301\240D\243\034\303\034\n\320\006]\274~+yH\347\315\031\350\235\023.\207\206\376\346.1\006\210\356\213B\226\3247\331\220Bs&IL\362m\203\252ma\275\003" signing_key_seq_num: 0 } } request_id { client_id: "9c65ab9956084620ba0b48b047897dba" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
W20260928 14:55:49.005594 1255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.116.129:40603 (error 108)
I20260928 14:55:49.005753 1794 pending_rounds.cc:68] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334327701369544704 op_type: NO_OP noop_request { }
I20260928 14:55:49.005795 1794 raft_consensus.cc:2866] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260928 14:55:49.005824 1794 raft_consensus.cc:2258] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:55:49.005990 1795 ts_tablet_manager.cc:1918] T 6b5801ca5d24491193ce95a473dc06da P 212b9b577ee04d11a86a9f319911ba39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:55:49.006059 1795 log.cc:1196] T 6b5801ca5d24491193ce95a473dc06da P 212b9b577ee04d11a86a9f319911ba39: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/6b5801ca5d24491193ce95a473dc06da
I20260928 14:55:49.006349 1795 ts_tablet_manager.cc:1939] T 6b5801ca5d24491193ce95a473dc06da P 212b9b577ee04d11a86a9f319911ba39: Deleting consensus metadata
I20260928 14:55:49.006623 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 1472
I20260928 14:55:49.010931 1770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:49.010786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:40603 (local address 127.0.0.1:56658)
0928 14:55:49.010846 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:49.010888 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.129:40603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 14:55:49.011044 1794 ts_tablet_manager.cc:1905] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 14:55:49.011148 1255 catalog_manager.cc:4584] TS 7647f0c709974cfa854d509ee77ae2ff (127.20.116.129:40603): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b5801ca5d24491193ce95a473dc06da: Network error: Client connection negotiation failed: client connection to 127.20.116.129:40603: connect: Connection refused (error 111)
I20260928 14:55:49.012987 1794 ts_tablet_manager.cc:1918] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:55:49.013069 1794 log.cc:1196] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/6b5801ca5d24491193ce95a473dc06da
I20260928 14:55:49.013337 1794 ts_tablet_manager.cc:1939] T 6b5801ca5d24491193ce95a473dc06da P 595b4cbd51a74962a8ec4088e07fd2bb: Deleting consensus metadata
I20260928 14:55:49.013875 1254 catalog_manager.cc:4841] TS 595b4cbd51a74962a8ec4088e07fd2bb (127.20.116.131:37953): tablet 6b5801ca5d24491193ce95a473dc06da (table test-table [id=505b109c1e424a9ba4266c6c6f252bc8]) successfully deleted
I20260928 14:55:49.015720 1770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:49.015601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:35027 (local address 127.0.0.1:58702)
0928 14:55:49.015658 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:49.015683 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.130:35027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 14:55:49.015873 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 1607
I20260928 14:55:49.021908 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 1236
I20260928 14:55:49.027808 20946 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 14:55:49.372411 20946 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 14:55:49.372498 20946 mini_ranger.cc:66] Stopping Ranger...
I20260928 14:55:49.716820 20946 mini_ranger.cc:68] Stopped Ranger
2026-09-28 14:55:49.717 UTC [32764] LOG: received smart shutdown request
2026-09-28 14:55:49.721 UTC [32764] LOG: background worker "logical replication launcher" (PID 305) exited with exit code 1
2026-09-28 14:55:49.722 UTC [300] LOG: shutting down
2026-09-28 14:55:49.895 UTC [32764] LOG: database system is shut down
2026-09-28T14:55:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49344 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:55:50 dist-test-slave-pxl6 krb5kdc[1806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:55:50 dist-test-slave-pxl6 krb5kdc[1806](info): set up 2 sockets
Sep 28 14:55:50 dist-test-slave-pxl6 krb5kdc[1806](info): commencing operation
krb5kdc: starting...
W20260928 14:55:51.051543 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:55:51 dist-test-slave-pxl6 krb5kdc[1806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:55:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:55:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:55:51.174311 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:42221
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:43045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:42221,127.20.116.189:46023,127.20.116.188:46345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:55:51.259016 1819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:51.259241 1819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:51.259265 1819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:51.260371 1819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:55:51.260407 1819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:51.260421 1819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:51.260434 1819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:55:51.260447 1819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:55:51.261911 1819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:43045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:42221,127.20.116.189:46023,127.20.116.188:46345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:42221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:51.262121 1819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:51.262305 1819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:51.267211 1826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:51.267216 1828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:51.267349 1819 server_base.cc:1034] running on GCE node
W20260928 14:55:51.267234 1825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:51.267865 1819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:51.268191 1819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:51.269351 1819 hybrid_clock.cc:648] HybridClock initialized: now 1790607351269337 us; error 33 us; skew 500 ppm
Sep 28 14:55:51 dist-test-slave-pxl6 krb5kdc[1806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607351, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:51.278024 1819 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:55:51.278620 1819 webserver.cc:460] Webserver started at http://127.20.116.190:41721/ using document root <none> and password file <none>
I20260928 14:55:51.278882 1819 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:51.278952 1819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:55:51.279100 1819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:55:51.280227 1819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "2faf98126e524ac092997bc7d071ab97"
format_stamp: "Formatted at 2026-09-28 14:55:51 on dist-test-slave-pxl6"
server_key: "7970a081ec426fe464c398a80e26c1ff"
server_key_iv: "a34d606706ba0020ce2d84aa07d030e6"
server_key_version: "encryptionkey@0"
I20260928 14:55:51.280587 1819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "2faf98126e524ac092997bc7d071ab97"
format_stamp: "Formatted at 2026-09-28 14:55:51 on dist-test-slave-pxl6"
server_key: "7970a081ec426fe464c398a80e26c1ff"
server_key_iv: "a34d606706ba0020ce2d84aa07d030e6"
server_key_version: "encryptionkey@0"
I20260928 14:55:51.281977 1819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:55:51.283052 1835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:51.283370 1819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:55:51.283481 1819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "2faf98126e524ac092997bc7d071ab97"
format_stamp: "Formatted at 2026-09-28 14:55:51 on dist-test-slave-pxl6"
server_key: "7970a081ec426fe464c398a80e26c1ff"
server_key_iv: "a34d606706ba0020ce2d84aa07d030e6"
server_key_version: "encryptionkey@0"
I20260928 14:55:51.283557 1819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:51.295969 1819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:51.298489 1819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:51.298704 1819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:51.303128 1819 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:42221
I20260928 14:55:51.303175 1887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:42221 every 8 connection(s)
I20260928 14:55:51.303557 1819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:55:51.304436 1888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:55:51.306041 1888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42221 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:55:51.309638 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 1819
I20260928 14:55:51.309791 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:55:51.310004 20946 external_mini_cluster.cc:1428] Setting key 535a8aabc66845ce4ee9b282240cebd5
Sep 28 14:55:51 dist-test-slave-pxl6 krb5kdc[1806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607351, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:55:51.317826 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.306802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:42221 (local address 127.0.0.1:38306)
0928 14:55:51.306977 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.307012 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:51.307463 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:51.307733 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:51.307735 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:51.308025 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:51.308409 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:51.308415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.309632 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.309633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:51.310879 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:51.310883 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.311458 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.311460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:51.311552 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.312197 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:51.312207 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:51.313889 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:51.316191 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:51.316205 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:51.316207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:51.316416 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:51.316717 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:51.316717 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:51.316718 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:51.316828 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:51.317322 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:51.317328 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:51.317392 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 14:55:51.317481 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 14:55:51.317624 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"spinlock_wait_cycles":7936,"thread_start_us":91,"threads_started":1}
I20260928 14:55:51.317902 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.306803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38306 (local address 127.20.116.190:42221)
0928 14:55:51.307060 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:51.307064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:51.307089 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:55:51.307602 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:51.307604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:51.307623 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:51.307703 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:51.308565 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.309507 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.310972 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.311340 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.311485 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.313991 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:51.314001 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:51.314002 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:51.314011 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:51.316085 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.316504 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.316508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.316509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.316640 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.316896 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.316897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.316897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.317223 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:51.317345 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:51.317543 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 14:55:51.317719 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":219,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for kudu/127.20.116.189@KRBTEST.COM; defaulting to no policy
I20260928 14:55:51.318689 1888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:55:51.319123 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.318980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:46023 (local address 127.0.0.1:41808)
0928 14:55:51.319055 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.319102 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.189:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 14:55:51.319294 1837 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.116.189:46023: connect: Connection refused (error 111)
I20260928 14:55:51.319888 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.319804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:46023 (local address 127.0.0.1:41820)
0928 14:55:51.319845 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.319870 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.189:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 14:55:51.320000 1888 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.189:46023: Network error: Client connection negotiation failed: client connection to 127.20.116.189:46023: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.189.keytab.
Entry for principal kudu/127.20.116.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.189.keytab.
I20260928 14:55:51.337402 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.189:46023
--webserver_interface=127.20.116.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.189.keytab
--principal=kudu/127.20.116.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:43045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:42221,127.20.116.189:46023,127.20.116.188:46345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:55:51.369251 1888 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } attempt: 1
I20260928 14:55:51.369738 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.369606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:46023 (local address 127.0.0.1:41828)
0928 14:55:51.369671 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.369707 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.189:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 14:55:51.370102 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.370009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:46023 (local address 127.0.0.1:41834)
0928 14:55:51.370052 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.370085 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.189:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 14:55:51.370185 1888 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.189:46023: Network error: Client connection negotiation failed: client connection to 127.20.116.189:46023: connect: Connection refused (error 111)
W20260928 14:55:51.422519 1895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:51.422870 1895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:51.422910 1895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:51.424180 1895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:55:51.424252 1895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:51.424273 1895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:51.424294 1895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:55:51.424325 1895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:55:51.425935 1895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:43045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:42221,127.20.116.189:46023,127.20.116.188:46345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.189.keytab
--principal=kudu/127.20.116.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.189:46023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.116.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:51.426229 1895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:51.426465 1895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:51.431396 1900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:51.431396 1901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:51.431459 1903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:51.431866 1895 server_base.cc:1034] running on GCE node
I20260928 14:55:51.432148 1895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:51.432446 1895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:51.433614 1895 hybrid_clock.cc:648] HybridClock initialized: now 1790607351433600 us; error 34 us; skew 500 ppm
Sep 28 14:55:51 dist-test-slave-pxl6 krb5kdc[1806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607351, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:51.436259 1895 init.cc:377] Logged in from keytab as kudu/127.20.116.189@KRBTEST.COM (short username kudu)
I20260928 14:55:51.436736 1895 webserver.cc:460] Webserver started at http://127.20.116.189:43727/ using document root <none> and password file <none>
I20260928 14:55:51.436986 1895 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:51.437053 1895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:55:51.437192 1895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:55:51.438331 1888 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } attempt: 2
I20260928 14:55:51.438298 1895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-1/data/instance:
uuid: "97e26782451a405cac09fb8c544bc34a"
format_stamp: "Formatted at 2026-09-28 14:55:51 on dist-test-slave-pxl6"
server_key: "22688d058ba5f8f82bce1be4c42657e5"
server_key_iv: "5ee93ac02e0669ea5d5ecdc1567405d0"
server_key_version: "encryptionkey@0"
I20260928 14:55:51.438731 1895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-1/wal/instance:
uuid: "97e26782451a405cac09fb8c544bc34a"
format_stamp: "Formatted at 2026-09-28 14:55:51 on dist-test-slave-pxl6"
server_key: "22688d058ba5f8f82bce1be4c42657e5"
server_key_iv: "5ee93ac02e0669ea5d5ecdc1567405d0"
server_key_version: "encryptionkey@0"
I20260928 14:55:51.438822 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.438666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:46023 (local address 127.0.0.1:41842)
0928 14:55:51.438734 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.438780 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.189:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 14:55:51.439219 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.439096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:46023 (local address 127.0.0.1:41848)
0928 14:55:51.439165 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.439195 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.189:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 14:55:51.439352 1888 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.189:46023: Network error: Client connection negotiation failed: client connection to 127.20.116.189:46023: connect: Connection refused (error 111)
I20260928 14:55:51.440267 1895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:55:51.441129 1910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:51.441325 1895 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:55:51.441417 1895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-1/wal
uuid: "97e26782451a405cac09fb8c544bc34a"
format_stamp: "Formatted at 2026-09-28 14:55:51 on dist-test-slave-pxl6"
server_key: "22688d058ba5f8f82bce1be4c42657e5"
server_key_iv: "5ee93ac02e0669ea5d5ecdc1567405d0"
server_key_version: "encryptionkey@0"
I20260928 14:55:51.441493 1895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:51.461974 1895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:51.464519 1895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:51.464634 1895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:51.468830 1895 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.189:46023
I20260928 14:55:51.468866 1962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.189:46023 every 8 connection(s)
I20260928 14:55:51.469246 1895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-1/data/info.pb
I20260928 14:55:51.470218 1964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:55:51.471917 1964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42221 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:55:51.472755 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 1895
I20260928 14:55:51.472856 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-1/wal/instance
I20260928 14:55:51.473053 20946 external_mini_cluster.cc:1428] Setting key 0842a72fa18fd2d201e431ceee0c7dcf
Sep 28 14:55:51 dist-test-slave-pxl6 krb5kdc[1806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607351, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.189@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.116.188@KRBTEST.COM; defaulting to no policy
I20260928 14:55:51.481745 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.472689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38310 (local address 127.20.116.190:42221)
0928 14:55:51.472742 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:51.472744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:51.473047 (+ 303us) server_negotiation.cc:407] Connection header received
0928 14:55:51.473619 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:51.473620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:51.473637 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:51.473685 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:51.474360 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.474913 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.475922 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.476132 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.476230 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.478838 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:51.478840 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:51.478840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:51.478850 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:51.480309 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.480811 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.480812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.480813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.480864 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.481171 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.481172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.481172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.481288 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:51.481344 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:51.481605 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 14:55:51.481677 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:55:51.481945 1966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.472676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:42221 (local address 127.0.0.1:38310)
0928 14:55:51.472905 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.472954 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:51.473501 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:51.473786 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:51.473790 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:51.474051 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:51.474256 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:51.474260 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.475013 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.475015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:51.475834 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:51.475837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.476239 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.476241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:51.476377 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.476930 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:51.476938 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:51.478703 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:51.480396 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:51.480406 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:51.480410 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:51.480721 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:51.480931 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:51.480932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:51.480933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:51.481095 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:51.481352 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:51.481359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:51.481490 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 14:55:51.481581 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 14:55:51.481715 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":91,"threads_started":1}
I20260928 14:55:51.482640 1964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.188@KRBTEST.COM" created.
Sep 28 14:55:51 dist-test-slave-pxl6 krb5kdc[1806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607351, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.189@KRBTEST.COM for kudu/127.20.116.189@KRBTEST.COM
I20260928 14:55:51.490581 1966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.482915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:46023 (local address 127.0.0.1:41862)
0928 14:55:51.482962 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.482973 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:51.483016 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:51.483262 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:51.483262 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:51.483382 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:51.483435 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:51.483436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.484606 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.484607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:51.485106 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:51.485107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.485596 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.485598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:51.485685 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.486151 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:51.486158 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:51.487716 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:51.489529 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:51.489530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:51.489530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:51.489691 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:51.489902 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:51.489903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:51.489903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:51.489956 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:51.490230 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:51.490231 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:51.490269 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 14:55:51.490334 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 14:55:51.490432 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 14:55:51.490646 1968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.482969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41862 (local address 127.20.116.189:46023)
0928 14:55:51.483132 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:51.483136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:51.483153 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:55:51.483172 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:51.483174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:51.483186 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:51.483256 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:51.483515 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.484503 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.485189 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.485499 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.485586 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.487794 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:51.487805 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:51.487806 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:51.487813 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:51.489428 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.489779 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.489783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.489783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.489839 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.490020 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.490021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.490022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.490164 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:51.490253 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:51.490362 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 14:55:51.490487 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260928 14:55:51.491204 1964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:55:51.491554 1966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.491437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:46345 (local address 127.0.0.1:35070)
0928 14:55:51.491481 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.491526 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:46345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 14:55:51.491711 1914 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.116.188:46345: connect: Connection refused (error 111)
I20260928 14:55:51.492084 1966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.491993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:46345 (local address 127.0.0.1:35074)
0928 14:55:51.492040 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.492063 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:46345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 14:55:51.492187 1964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.188:46345: Network error: Client connection negotiation failed: client connection to 127.20.116.188:46345: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.188.keytab.
Entry for principal kudu/127.20.116.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.188.keytab.
I20260928 14:55:51.499678 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.188:46345
--webserver_interface=127.20.116.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.188.keytab
--principal=kudu/127.20.116.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:43045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:42221,127.20.116.189:46023,127.20.116.188:46345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:55:51.530575 1888 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } attempt: 3
Sep 28 14:55:51 dist-test-slave-pxl6 krb5kdc[1806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607351, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for kudu/127.20.116.189@KRBTEST.COM
I20260928 14:55:51.538062 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.530973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:46023 (local address 127.0.0.1:41872)
0928 14:55:51.531046 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.531058 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:51.531124 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:51.531267 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:51.531269 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:51.531446 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:51.531511 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:51.531512 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.532192 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.532194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:51.532732 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:51.532734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.533147 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.533148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:51.533230 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.533704 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:51.533709 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:51.534988 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:51.537067 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:51.537068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:51.537068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:51.537256 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:51.537474 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:51.537475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:51.537475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:51.537527 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:51.537812 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:51.537812 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:51.537854 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 14:55:51.537907 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 14:55:51.537989 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 14:55:51.538127 1968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.531062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41872 (local address 127.20.116.189:46023)
0928 14:55:51.531127 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:51.531129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:51.531150 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:55:51.531182 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:51.531183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:51.531195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:51.531261 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:51.531594 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.532111 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.532824 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.533042 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.533145 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.535088 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:51.535090 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:51.535091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:51.535100 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:51.536967 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.537333 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.537334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.537335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.537406 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.537584 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.537584 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.537585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.537731 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:51.537801 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:51.537950 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 14:55:51.538037 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:55:51.538587 1888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:55:51.538935 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.538832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:46345 (local address 127.0.0.1:35080)
0928 14:55:51.538880 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.538908 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:46345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 14:55:51.539278 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.539201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:46345 (local address 127.0.0.1:35082)
0928 14:55:51.539240 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.539262 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:46345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 14:55:51.539386 1888 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.188:46345: Network error: Client connection negotiation failed: client connection to 127.20.116.188:46345: connect: Connection refused (error 111)
I20260928 14:55:51.541386 1964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } attempt: 1
I20260928 14:55:51.541749 1966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.541645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:46345 (local address 127.0.0.1:35094)
0928 14:55:51.541694 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.541721 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:46345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 14:55:51.542094 1966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.542008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:46345 (local address 127.0.0.1:35096)
0928 14:55:51.542051 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.542074 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:46345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 14:55:51.542184 1964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.188:46345: Network error: Client connection negotiation failed: client connection to 127.20.116.188:46345: connect: Connection refused (error 111)
I20260928 14:55:51.570562 1888 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } attempt: 1
I20260928 14:55:51.571074 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.570925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:46345 (local address 127.0.0.1:35102)
0928 14:55:51.570994 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.571039 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:46345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 14:55:51.571451 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.571353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:46345 (local address 127.0.0.1:35104)
0928 14:55:51.571398 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.571428 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:46345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 14:55:51.571565 1888 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.188:46345: Network error: Client connection negotiation failed: client connection to 127.20.116.188:46345: connect: Connection refused (error 111)
W20260928 14:55:51.585477 1971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:51.585704 1971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:51.585729 1971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:51.586967 1971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:55:51.587009 1971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:51.587024 1971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:51.587064 1971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:55:51.587086 1971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:55:51.588661 1971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:43045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:42221,127.20.116.189:46023,127.20.116.188:46345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.188.keytab
--principal=kudu/127.20.116.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.188:46345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.116.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:51.588920 1971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:51.589172 1971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:51.594075 1976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:51.594079 1979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:51.594100 1971 server_base.cc:1034] running on GCE node
W20260928 14:55:51.594079 1977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:51.594522 1971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:51.594885 1971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:51.596043 1971 hybrid_clock.cc:648] HybridClock initialized: now 1790607351596028 us; error 30 us; skew 500 ppm
Sep 28 14:55:51 dist-test-slave-pxl6 krb5kdc[1806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607351, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:51.598491 1971 init.cc:377] Logged in from keytab as kudu/127.20.116.188@KRBTEST.COM (short username kudu)
I20260928 14:55:51.598982 1971 webserver.cc:460] Webserver started at http://127.20.116.188:42865/ using document root <none> and password file <none>
I20260928 14:55:51.599241 1971 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:51.599311 1971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:55:51.599453 1971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:55:51.600564 1971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-2/data/instance:
uuid: "2208f6149e594534ab1fd14720f07fef"
format_stamp: "Formatted at 2026-09-28 14:55:51 on dist-test-slave-pxl6"
server_key: "4d3c3243616b8cb151dea0e1fc6dc8fc"
server_key_iv: "db1c747303a116f464049654c658b18f"
server_key_version: "encryptionkey@0"
I20260928 14:55:51.600924 1971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-2/wal/instance:
uuid: "2208f6149e594534ab1fd14720f07fef"
format_stamp: "Formatted at 2026-09-28 14:55:51 on dist-test-slave-pxl6"
server_key: "4d3c3243616b8cb151dea0e1fc6dc8fc"
server_key_iv: "db1c747303a116f464049654c658b18f"
server_key_version: "encryptionkey@0"
I20260928 14:55:51.602373 1971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 14:55:51.603245 1986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:51.603454 1971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:55:51.603556 1971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-2/wal
uuid: "2208f6149e594534ab1fd14720f07fef"
format_stamp: "Formatted at 2026-09-28 14:55:51 on dist-test-slave-pxl6"
server_key: "4d3c3243616b8cb151dea0e1fc6dc8fc"
server_key_iv: "db1c747303a116f464049654c658b18f"
server_key_version: "encryptionkey@0"
I20260928 14:55:51.603636 1971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:51.610327 1964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } attempt: 2
I20260928 14:55:51.610827 1966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.610697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:46345 (local address 127.0.0.1:35118)
0928 14:55:51.610757 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.610795 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:46345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 14:55:51.611172 1966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.611077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:46345 (local address 127.0.0.1:35128)
0928 14:55:51.611124 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.611148 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:46345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 14:55:51.611254 1964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.188:46345: Network error: Client connection negotiation failed: client connection to 127.20.116.188:46345: connect: Connection refused (error 111)
I20260928 14:55:51.615056 1971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:51.617287 1971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:51.617449 1971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:51.621694 1971 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.188:46345
I20260928 14:55:51.621726 2038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.188:46345 every 8 connection(s)
I20260928 14:55:51.622186 1971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-2/data/info.pb
I20260928 14:55:51.623386 2039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:55:51.624541 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 1971
I20260928 14:55:51.624671 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607192794321-20946-0/minicluster-data/master-2/wal/instance
I20260928 14:55:51.624876 20946 external_mini_cluster.cc:1428] Setting key 671618694b41a69b7bf48acbd647e2d6
I20260928 14:55:51.625505 2039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42221 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:55:51 dist-test-slave-pxl6 krb5kdc[1806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607351, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.188@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
Sep 28 14:55:51 dist-test-slave-pxl6 krb5kdc[1806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.188@KRBTEST.COM
I20260928 14:55:51.636181 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.626248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38320 (local address 127.20.116.190:42221)
0928 14:55:51.626304 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:51.626307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:51.626513 (+ 206us) server_negotiation.cc:407] Connection header received
0928 14:55:51.626679 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:51.626680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:51.626691 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:51.626759 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:51.627355 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.627954 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.629241 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.629535 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.629598 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.632650 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:51.632651 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:51.632652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:51.632659 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:51.634424 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.634989 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.634991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.634991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.635058 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.635416 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.635417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.635417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.635569 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:51.635653 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:51.635990 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 14:55:51.636076 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:55:51.636380 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.626191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:42221 (local address 127.0.0.1:38320)
0928 14:55:51.626389 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.626423 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:51.626512 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:51.626789 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:51.626792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:51.627099 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:51.627263 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:51.627268 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.628047 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.628048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:51.629118 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:51.629122 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.629662 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.629663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:51.629792 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.630847 (+ 1055us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:51.630886 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:51.632550 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:51.634523 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:51.634588 (+ 65us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:51.634590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:51.634916 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:51.635185 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:51.635186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:51.635187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:51.635336 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:51.635649 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:51.635656 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:51.635849 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 14:55:51.635928 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 14:55:51.636109 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":99,"threads_started":1}
I20260928 14:55:51.636384 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.625462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35142 (local address 127.20.116.188:46345)
0928 14:55:51.625639 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:51.625649 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:51.625683 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:55:51.626325 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:51.626334 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:51.626398 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:51.626580 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:51.627192 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.628171 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.629283 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.629758 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.629819 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.633177 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:51.633185 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:51.633186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:51.633193 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:51.635107 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.635527 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.635530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.635530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.635584 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.635777 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.635777 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.635777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.635909 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:51.636076 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:51.636178 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 14:55:51.636259 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
I20260928 14:55:51.637069 2039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:55:51 dist-test-slave-pxl6 krb5kdc[1806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607351, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.188@KRBTEST.COM for kudu/127.20.116.189@KRBTEST.COM
Sep 28 14:55:51 dist-test-slave-pxl6 krb5kdc[1806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:55:51.645268 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.637465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:46023 (local address 127.0.0.1:41880)
0928 14:55:51.637550 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.637558 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:51.637590 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:51.637721 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:51.637722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:51.637908 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:51.637966 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:51.637968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.638900 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.638901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:51.639702 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:51.639704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.640212 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.640213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:51.640283 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.640744 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:51.640748 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:51.642065 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:51.644109 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:51.644111 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:51.644111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:51.644380 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:51.644585 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:51.644586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:51.644586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:51.644658 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:51.644950 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:51.644951 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:51.644995 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 14:55:51.645072 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 14:55:51.645162 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 14:55:51.645323 1968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.637559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41880 (local address 127.20.116.189:46023)
0928 14:55:51.637605 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:51.637607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:51.637613 (+ 6us) server_negotiation.cc:407] Connection header received
0928 14:55:51.637646 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:51.637647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:51.637654 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:51.637713 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:51.638041 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.638828 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.639829 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.640127 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.640196 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.642153 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:51.642154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:51.642155 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:51.642162 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:51.644029 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.644451 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.644452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.644453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.644508 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.644723 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.644723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.644724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.644888 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:51.644950 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:51.645097 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 14:55:51.645179 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:55:51.645658 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.637392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38322 (local address 127.20.116.190:42221)
0928 14:55:51.637442 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:51.637444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:51.637453 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:55:51.637475 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:51.637475 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:51.637484 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:51.637537 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:51.638171 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.638878 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.639826 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.640124 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.640195 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.642861 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:51.642862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:51.642862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:51.642869 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:51.644278 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.644651 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.644652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.644652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.644708 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.644893 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.644893 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.644894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.645028 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:51.645213 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:51.645251 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 14:55:51.645597 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 14:55:51.645991 2039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:55:51.646728 1888 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } attempt: 2
Sep 28 14:55:51 dist-test-slave-pxl6 krb5kdc[1806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.189@KRBTEST.COM
Sep 28 14:55:51 dist-test-slave-pxl6 krb5kdc[1806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607351, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.188@KRBTEST.COM for kudu/127.20.116.188@KRBTEST.COM
Sep 28 14:55:51 dist-test-slave-pxl6 krb5kdc[1806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607351, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for kudu/127.20.116.188@KRBTEST.COM
I20260928 14:55:51.655253 1968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.646330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41892 (local address 127.20.116.189:46023)
0928 14:55:51.646365 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:51.646367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:51.646374 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:55:51.646394 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:51.646395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:51.646406 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:51.646464 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:51.647133 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.647755 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.648599 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.648801 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.648889 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.651665 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:51.651667 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:51.651667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:51.651674 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:51.653653 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.654042 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.654043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.654043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.654107 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.654363 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.654364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.654364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.654522 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:51.654795 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:51.655027 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 14:55:51.655142 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:55:51.655385 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.646282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:46345 (local address 127.0.0.1:35156)
0928 14:55:51.646361 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.646370 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:51.646410 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:51.646514 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:51.646515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:51.646685 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:51.646723 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:51.646725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.647610 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.647611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:51.648315 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:51.648318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.649266 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.649267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:51.649322 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.650519 (+ 1197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:51.650525 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:51.652061 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:51.653925 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:51.653926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:51.653927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:51.654255 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:51.654600 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:51.654601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:51.654602 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:51.654678 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:51.654990 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:51.654991 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:51.655036 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 14:55:51.655106 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:55:51.655259 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 14:55:51.656003 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.646322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35156 (local address 127.20.116.188:46345)
0928 14:55:51.646361 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:51.646364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:51.646399 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:55:51.646453 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:51.646454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:51.646463 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:51.646512 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:51.646803 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.647531 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.648387 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.649191 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.649259 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.652148 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:51.652148 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:51.652149 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:51.652154 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:51.653848 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.654328 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.654329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.654330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.654392 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.654751 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.654752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.654752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.654903 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:51.654989 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:51.655166 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 14:55:51.655904 (+ 738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":20}
I20260928 14:55:51.656945 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.646958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:46345 (local address 127.0.0.1:35162)
0928 14:55:51.647002 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.647011 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:51.647044 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:51.647971 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:51.647973 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:51.648115 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:51.648150 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:51.648151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.649068 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.649069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:51.649795 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:51.649798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.650156 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.650158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:51.650220 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.651864 (+ 1644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:51.651869 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:51.653250 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:51.655583 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:51.655584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:51.655585 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:51.655778 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:51.656092 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:51.656093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:51.656094 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:51.656164 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:51.656608 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:51.656609 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:51.656657 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 14:55:51.656727 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:55:51.656849 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 14:55:51.657069 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:55:51.657264 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.647007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35162 (local address 127.20.116.188:46345)
0928 14:55:51.647858 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:51.647859 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:51.647864 (+ 5us) server_negotiation.cc:407] Connection header received
0928 14:55:51.647879 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:51.647879 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:51.647887 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:51.647973 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:51.648247 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.648975 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.649888 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.650082 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.650150 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.653366 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:51.653367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:51.653368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:51.653374 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:51.655451 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.655847 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.655848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.655848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.655916 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.656239 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.656240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.656240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.656546 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:51.656616 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:51.656763 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 14:55:51.656852 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":100,"threads_started":1}
I20260928 14:55:51.657959 2039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef: Bootstrap starting.
I20260928 14:55:51.658885 2039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef: Neither blocks nor log segments found. Creating new log.
I20260928 14:55:51.659237 2039 log.cc:824] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef: Log is configured to *not* fsync() on all Append() calls
I20260928 14:55:51.659545 1888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97: Bootstrap starting.
I20260928 14:55:51.660125 2039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef: No bootstrap required, opened a new log
I20260928 14:55:51.660317 1888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97: Neither blocks nor log segments found. Creating new log.
I20260928 14:55:51.660640 1888 log.cc:824] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97: Log is configured to *not* fsync() on all Append() calls
I20260928 14:55:51.662128 2039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2faf98126e524ac092997bc7d071ab97" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42221 } } peers { permanent_uuid: "97e26782451a405cac09fb8c544bc34a" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } } peers { permanent_uuid: "2208f6149e594534ab1fd14720f07fef" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } }
I20260928 14:55:51.662312 2039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:55:51.662362 2039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2208f6149e594534ab1fd14720f07fef, State: Initialized, Role: FOLLOWER
I20260928 14:55:51.662504 2039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2faf98126e524ac092997bc7d071ab97" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42221 } } peers { permanent_uuid: "97e26782451a405cac09fb8c544bc34a" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } } peers { permanent_uuid: "2208f6149e594534ab1fd14720f07fef" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } }
I20260928 14:55:51.662940 2039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:55:51.664887 2054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2faf98126e524ac092997bc7d071ab97" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42221 } } peers { permanent_uuid: "97e26782451a405cac09fb8c544bc34a" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } } peers { permanent_uuid: "2208f6149e594534ab1fd14720f07fef" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } } }
I20260928 14:55:51.665010 2054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:55:51.666028 1888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97: No bootstrap required, opened a new log
I20260928 14:55:51.665968 1971 master_runner.cc:186] Error getting master registration for 127.20.116.190:42221: OK, instance_id { permanent_uuid: "2faf98126e524ac092997bc7d071ab97" instance_seqno: 1790607351301850 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:55:51.668612 1971 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:55:51.668962 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.658890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38324 (local address 127.20.116.190:42221)
0928 14:55:51.658976 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:51.658978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:51.658985 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:55:51.659006 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:51.659006 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:51.659015 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:51.659064 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:51.662133 (+ 3069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.662828 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.663858 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.664172 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.664215 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.665669 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:51.665670 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:51.665670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:51.665677 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:51.667466 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.668007 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.668008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.668008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.668066 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.668318 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.668319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.668319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.668452 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:51.668529 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:51.668751 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 14:55:51.668848 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 14:55:51.668972 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.658520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35164 (local address 127.20.116.188:46345)
0928 14:55:51.658744 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:51.658746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:51.659190 (+ 444us) server_negotiation.cc:407] Connection header received
0928 14:55:51.659256 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:51.659257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:51.659270 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:51.659334 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:51.660567 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.661260 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.662103 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.662381 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.662440 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.664285 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:51.664286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:51.664287 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:51.664294 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:51.667472 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.668007 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.668008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.668008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.668066 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.668314 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.668315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.668315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.668466 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:51.668559 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:51.668751 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 14:55:51.668870 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
I20260928 14:55:51.668411 1888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2faf98126e524ac092997bc7d071ab97" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42221 } } peers { permanent_uuid: "97e26782451a405cac09fb8c544bc34a" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } } peers { permanent_uuid: "2208f6149e594534ab1fd14720f07fef" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } }
I20260928 14:55:51.670250 1888 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:55:51.670306 1888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2faf98126e524ac092997bc7d071ab97, State: Initialized, Role: FOLLOWER
I20260928 14:55:51.670481 1888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2faf98126e524ac092997bc7d071ab97" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42221 } } peers { permanent_uuid: "97e26782451a405cac09fb8c544bc34a" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } } peers { permanent_uuid: "2208f6149e594534ab1fd14720f07fef" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } }
I20260928 14:55:51.670626 1968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.658931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41902 (local address 127.20.116.189:46023)
0928 14:55:51.658972 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:51.658974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:51.659343 (+ 369us) server_negotiation.cc:407] Connection header received
0928 14:55:51.659440 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:51.659440 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:51.659451 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:51.659479 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:51.660118 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.660878 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.661876 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.662211 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.662297 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.664683 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:51.664685 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:51.664685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:51.664694 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:51.669400 (+ 4706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.669771 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.669772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.669773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.669842 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.670031 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.670032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.670032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.670182 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:51.670294 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:51.670387 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 14:55:51.670478 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:55:51.670827 2066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2faf98126e524ac092997bc7d071ab97" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42221 } } peers { permanent_uuid: "97e26782451a405cac09fb8c544bc34a" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } } peers { permanent_uuid: "2208f6149e594534ab1fd14720f07fef" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } } }
I20260928 14:55:51.670910 1888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:55:51.670913 2066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 14:55:51.671705 2065 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:55:51.671764 2065 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 14:55:51.672941 2078 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:55:51.673005 2078 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:55:51.674026 1819 master_runner.cc:186] Error getting master registration for 127.20.116.189:46023: OK, instance_id { permanent_uuid: "97e26782451a405cac09fb8c544bc34a" instance_seqno: 1790607351467618 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:55:51.676186 1819 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:55:51.702409 1964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } attempt: 3
Sep 28 14:55:51 dist-test-slave-pxl6 krb5kdc[1806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607351, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.189@KRBTEST.COM for kudu/127.20.116.188@KRBTEST.COM
I20260928 14:55:51.710831 1966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.702790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:46345 (local address 127.0.0.1:35180)
0928 14:55:51.702857 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:51.702869 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:51.702918 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:51.703073 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:51.703074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:51.703232 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:51.703294 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:51.703296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.704011 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.704012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:51.704558 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:51.704561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.705391 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.705392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:51.705459 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.705923 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:51.705965 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:51.707464 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:51.709566 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:51.709567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:51.709568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:51.709801 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:51.710020 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:51.710021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:51.710021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:51.710083 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:51.710422 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:51.710424 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:51.710479 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 14:55:51.710575 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 14:55:51.710692 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 14:55:51.711279 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:51.702851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.20.116.188:46345)
0928 14:55:51.702911 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:51.702913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:51.702929 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:55:51.702965 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:51.702966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:51.702979 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:51.703061 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:51.703382 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.703924 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.705018 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:51.705293 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:51.705376 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:51.707571 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:51.707573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:51.707573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:51.707581 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:51.709467 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.709885 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.709887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.709888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.709948 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:51.710151 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:51.710152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:51.710152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:51.710313 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:51.710391 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:51.711043 (+ 652us) server_negotiation.cc:299] Negotiation successful
0928 14:55:51.711137 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 14:55:51.715376 1964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a: Bootstrap starting.
I20260928 14:55:51.716253 1964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a: Neither blocks nor log segments found. Creating new log.
I20260928 14:55:51.716626 1964 log.cc:824] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:55:51.717983 1964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a: No bootstrap required, opened a new log
I20260928 14:55:51.720183 1964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2faf98126e524ac092997bc7d071ab97" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42221 } } peers { permanent_uuid: "97e26782451a405cac09fb8c544bc34a" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } } peers { permanent_uuid: "2208f6149e594534ab1fd14720f07fef" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } }
I20260928 14:55:51.720376 1964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:55:51.720424 1964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e26782451a405cac09fb8c544bc34a, State: Initialized, Role: FOLLOWER
I20260928 14:55:51.720553 1964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2faf98126e524ac092997bc7d071ab97" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42221 } } peers { permanent_uuid: "97e26782451a405cac09fb8c544bc34a" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } } peers { permanent_uuid: "2208f6149e594534ab1fd14720f07fef" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } }
I20260928 14:55:51.720877 2081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2faf98126e524ac092997bc7d071ab97" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42221 } } peers { permanent_uuid: "97e26782451a405cac09fb8c544bc34a" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } } peers { permanent_uuid: "2208f6149e594534ab1fd14720f07fef" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } } }
I20260928 14:55:51.720948 1964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:55:51.721022 2081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a [sys.catalog]: This master's current role is: FOLLOWER
W20260928 14:55:51.722880 2092 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:55:51.722993 2092 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:55:51.725329 1895 master_runner.cc:263] No leader master found from master 97e26782451a405cac09fb8c544bc34a
I20260928 14:55:51.725378 1895 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 14:55:51.799624 1841 tablet.cc:2369] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:55:51.868124 1992 tablet.cc:2369] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:55:51.967159 1916 tablet.cc:2369] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:55:51.989398 2066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:55:51.989514 2066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2faf98126e524ac092997bc7d071ab97" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42221 } } peers { permanent_uuid: "97e26782451a405cac09fb8c544bc34a" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } } peers { permanent_uuid: "2208f6149e594534ab1fd14720f07fef" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } }
I20260928 14:55:51.989795 2066 leader_election.cc:288] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97e26782451a405cac09fb8c544bc34a (127.20.116.189:46023), 2208f6149e594534ab1fd14720f07fef (127.20.116.188:46345)
I20260928 14:55:51.990031 2013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2faf98126e524ac092997bc7d071ab97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2208f6149e594534ab1fd14720f07fef" is_pre_election: true
I20260928 14:55:51.990047 1937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2faf98126e524ac092997bc7d071ab97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e26782451a405cac09fb8c544bc34a" is_pre_election: true
I20260928 14:55:51.990187 2013 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2faf98126e524ac092997bc7d071ab97 in term 0.
I20260928 14:55:51.990203 1937 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2faf98126e524ac092997bc7d071ab97 in term 0.
I20260928 14:55:51.990404 1837 leader_election.cc:302] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2faf98126e524ac092997bc7d071ab97, 97e26782451a405cac09fb8c544bc34a; no voters:
I20260928 14:55:51.990566 2066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:55:51.990629 2066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:55:51.990658 2066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:55:51.991379 2066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2faf98126e524ac092997bc7d071ab97" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42221 } } peers { permanent_uuid: "97e26782451a405cac09fb8c544bc34a" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } } peers { permanent_uuid: "2208f6149e594534ab1fd14720f07fef" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } }
I20260928 14:55:51.991524 2066 leader_election.cc:288] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [CANDIDATE]: Term 1 election: Requested vote from peers 97e26782451a405cac09fb8c544bc34a (127.20.116.189:46023), 2208f6149e594534ab1fd14720f07fef (127.20.116.188:46345)
I20260928 14:55:51.991645 1937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2faf98126e524ac092997bc7d071ab97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e26782451a405cac09fb8c544bc34a"
I20260928 14:55:51.991686 2013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2faf98126e524ac092997bc7d071ab97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2208f6149e594534ab1fd14720f07fef"
I20260928 14:55:51.991743 1937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:55:51.991778 2013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:55:51.992444 1937 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2faf98126e524ac092997bc7d071ab97 in term 1.
I20260928 14:55:51.992542 2013 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2faf98126e524ac092997bc7d071ab97 in term 1.
I20260928 14:55:51.992707 1837 leader_election.cc:302] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2faf98126e524ac092997bc7d071ab97, 97e26782451a405cac09fb8c544bc34a; no voters:
I20260928 14:55:51.992820 2066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:55:51.992954 2066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [term 1 LEADER]: Becoming Leader. State: Replica: 2faf98126e524ac092997bc7d071ab97, State: Running, Role: LEADER
I20260928 14:55:51.993058 2066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2faf98126e524ac092997bc7d071ab97" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42221 } } peers { permanent_uuid: "97e26782451a405cac09fb8c544bc34a" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } } peers { permanent_uuid: "2208f6149e594534ab1fd14720f07fef" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } }
I20260928 14:55:51.993445 2094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2faf98126e524ac092997bc7d071ab97. Latest consensus state: current_term: 1 leader_uuid: "2faf98126e524ac092997bc7d071ab97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2faf98126e524ac092997bc7d071ab97" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42221 } } peers { permanent_uuid: "97e26782451a405cac09fb8c544bc34a" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } } peers { permanent_uuid: "2208f6149e594534ab1fd14720f07fef" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } } }
I20260928 14:55:51.993537 2094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [sys.catalog]: This master's current role is: LEADER
I20260928 14:55:51.993736 2096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:55:51.993937 2096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:55:51.995499 1937 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a [term 1 FOLLOWER]: Refusing update from remote peer 2faf98126e524ac092997bc7d071ab97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:55:51.995517 2013 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef [term 1 FOLLOWER]: Refusing update from remote peer 2faf98126e524ac092997bc7d071ab97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:55:51.995854 2066 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97e26782451a405cac09fb8c544bc34a" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:55:51.995965 2094 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2208f6149e594534ab1fd14720f07fef" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:55:51.997227 2081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2faf98126e524ac092997bc7d071ab97. Latest consensus state: current_term: 1 leader_uuid: "2faf98126e524ac092997bc7d071ab97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2faf98126e524ac092997bc7d071ab97" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42221 } } peers { permanent_uuid: "97e26782451a405cac09fb8c544bc34a" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } } peers { permanent_uuid: "2208f6149e594534ab1fd14720f07fef" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } } }
I20260928 14:55:51.997241 2054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2faf98126e524ac092997bc7d071ab97. Latest consensus state: current_term: 1 leader_uuid: "2faf98126e524ac092997bc7d071ab97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2faf98126e524ac092997bc7d071ab97" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42221 } } peers { permanent_uuid: "97e26782451a405cac09fb8c544bc34a" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } } peers { permanent_uuid: "2208f6149e594534ab1fd14720f07fef" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } } }
I20260928 14:55:51.997328 2054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:55:51.997335 2081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:55:51.998291 2066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2faf98126e524ac092997bc7d071ab97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2faf98126e524ac092997bc7d071ab97" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42221 } } peers { permanent_uuid: "97e26782451a405cac09fb8c544bc34a" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } } peers { permanent_uuid: "2208f6149e594534ab1fd14720f07fef" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } } }
I20260928 14:55:51.998315 2094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2faf98126e524ac092997bc7d071ab97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2faf98126e524ac092997bc7d071ab97" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42221 } } peers { permanent_uuid: "97e26782451a405cac09fb8c544bc34a" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } } peers { permanent_uuid: "2208f6149e594534ab1fd14720f07fef" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } } }
I20260928 14:55:51.998394 2066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [sys.catalog]: This master's current role is: LEADER
I20260928 14:55:51.998448 2097 mvcc.cc:205] Tried to move back new op lower bound from 7334327713770688512 to 7334327713764200448. Current Snapshot: MvccSnapshot[applied={T|T < 7334327713770688512}]
I20260928 14:55:51.998517 2081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2faf98126e524ac092997bc7d071ab97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2faf98126e524ac092997bc7d071ab97" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42221 } } peers { permanent_uuid: "97e26782451a405cac09fb8c544bc34a" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } } peers { permanent_uuid: "2208f6149e594534ab1fd14720f07fef" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } } }
I20260928 14:55:51.998621 2081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:55:51.998878 2096 catalog_manager.cc:1347] Generated new cluster ID: e68f3448533f4923a6a08b41931e2dd0
I20260928 14:55:51.998927 2096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:55:51.998925 2099 mvcc.cc:205] Tried to move back new op lower bound from 7334327713770688512 to 7334327713764200448. Current Snapshot: MvccSnapshot[applied={T|T < 7334327713770688512}]
I20260928 14:55:51.999006 2054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2faf98126e524ac092997bc7d071ab97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2faf98126e524ac092997bc7d071ab97" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42221 } } peers { permanent_uuid: "97e26782451a405cac09fb8c544bc34a" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 46023 } } peers { permanent_uuid: "2208f6149e594534ab1fd14720f07fef" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 46345 } } }
I20260928 14:55:51.999060 2054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:55:51.998400 2094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97 [sys.catalog]: This master's current role is: LEADER
I20260928 14:55:52.016530 2096 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:55:52.017189 2096 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:55:52.023660 2096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2faf98126e524ac092997bc7d071ab97: Generated new TSK 0
I20260928 14:55:52.023917 2096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:55:52.672297 2065 catalog_manager.cc:1259] Loaded cluster ID: e68f3448533f4923a6a08b41931e2dd0
I20260928 14:55:52.672376 2065 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef: loading cluster ID for follower catalog manager: success
I20260928 14:55:52.673305 2065 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef: acquiring CA information for follower catalog manager: success
I20260928 14:55:52.673632 2065 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2208f6149e594534ab1fd14720f07fef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:55:52.723512 2092 catalog_manager.cc:1259] Loaded cluster ID: e68f3448533f4923a6a08b41931e2dd0
I20260928 14:55:52.723640 2092 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a: loading cluster ID for follower catalog manager: success
I20260928 14:55:52.724510 2092 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a: acquiring CA information for follower catalog manager: success
I20260928 14:55:52.724874 2092 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 97e26782451a405cac09fb8c544bc34a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:55:53.085214 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 1819
I20260928 14:55:53.090902 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 1895
I20260928 14:55:53.096153 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 1971
2026-09-28T14:55:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3090 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:55:53 dist-test-slave-pxl6 krb5kdc[2113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:55:53 dist-test-slave-pxl6 krb5kdc[2113](info): set up 2 sockets
Sep 28 14:55:53 dist-test-slave-pxl6 krb5kdc[2113](info): commencing operation
krb5kdc: starting...
W20260928 14:55:54.147751 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:55:54 dist-test-slave-pxl6 krb5kdc[2113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:55:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:55:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:55:54.274497 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:33181
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:37321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:33181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:55:54.357924 2126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:54.358139 2126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:54.358161 2126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:54.359368 2126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:55:54.359411 2126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:54.359426 2126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:54.359437 2126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:55:54.359449 2126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:55:54.361090 2126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:37321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:33181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:33181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:54.361368 2126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:54.361611 2126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:54.366755 2134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:54.366747 2132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:54.366748 2131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:54.366943 2126 server_base.cc:1034] running on GCE node
I20260928 14:55:54.367229 2126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:54.367580 2126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:54.368758 2126 hybrid_clock.cc:648] HybridClock initialized: now 1790607354368740 us; error 49 us; skew 500 ppm
Sep 28 14:55:54 dist-test-slave-pxl6 krb5kdc[2113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607354, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:54.371471 2126 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:55:54.371960 2126 webserver.cc:460] Webserver started at http://127.20.116.190:43697/ using document root <none> and password file <none>
I20260928 14:55:54.372201 2126 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:54.372277 2126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:55:54.372444 2126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:55:54.373593 2126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "10803799cfee44bcb6e9dca381e52103"
format_stamp: "Formatted at 2026-09-28 14:55:54 on dist-test-slave-pxl6"
server_key: "9e7d7bd197604bf004354062c1adfc08"
server_key_iv: "156960178d4b810e468f3359a3c28d1d"
server_key_version: "encryptionkey@0"
I20260928 14:55:54.373970 2126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "10803799cfee44bcb6e9dca381e52103"
format_stamp: "Formatted at 2026-09-28 14:55:54 on dist-test-slave-pxl6"
server_key: "9e7d7bd197604bf004354062c1adfc08"
server_key_iv: "156960178d4b810e468f3359a3c28d1d"
server_key_version: "encryptionkey@0"
I20260928 14:55:54.375519 2126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:55:54.376295 2141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:54.376472 2126 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:55:54.376590 2126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "10803799cfee44bcb6e9dca381e52103"
format_stamp: "Formatted at 2026-09-28 14:55:54 on dist-test-slave-pxl6"
server_key: "9e7d7bd197604bf004354062c1adfc08"
server_key_iv: "156960178d4b810e468f3359a3c28d1d"
server_key_version: "encryptionkey@0"
I20260928 14:55:54.376667 2126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:54.391355 2126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:54.393653 2126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:54.393838 2126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:54.397945 2126 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:33181
I20260928 14:55:54.397997 2193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:33181 every 8 connection(s)
I20260928 14:55:54.398394 2126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:55:54.399390 2194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:55:54.399718 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 2126
I20260928 14:55:54.399828 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:55:54.400020 20946 external_mini_cluster.cc:1428] Setting key b45751fbbd4a61da2e1f6a48eb87d622
I20260928 14:55:54.403077 2194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103: Bootstrap starting.
I20260928 14:55:54.403800 2194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103: Neither blocks nor log segments found. Creating new log.
I20260928 14:55:54.404098 2194 log.cc:824] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103: Log is configured to *not* fsync() on all Append() calls
I20260928 14:55:54.404891 2194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103: No bootstrap required, opened a new log
Sep 28 14:55:54 dist-test-slave-pxl6 krb5kdc[2113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:55:54.407617 2194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10803799cfee44bcb6e9dca381e52103" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 33181 } }
I20260928 14:55:54.407788 2194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:55:54.407850 2194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10803799cfee44bcb6e9dca381e52103, State: Initialized, Role: FOLLOWER
I20260928 14:55:54.408007 2194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10803799cfee44bcb6e9dca381e52103" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 33181 } }
I20260928 14:55:54.408092 2194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:55:54.408134 2194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:55:54.408185 2194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:55:54.408910 2194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10803799cfee44bcb6e9dca381e52103" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 33181 } }
I20260928 14:55:54.409037 2194 leader_election.cc:302] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10803799cfee44bcb6e9dca381e52103; no voters:
I20260928 14:55:54.409276 2194 leader_election.cc:288] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:55:54.409332 2199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:55:54.409475 2199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103 [term 1 LEADER]: Becoming Leader. State: Replica: 10803799cfee44bcb6e9dca381e52103, State: Running, Role: LEADER
I20260928 14:55:54.409567 2194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:55:54.409644 2199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10803799cfee44bcb6e9dca381e52103" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 33181 } }
I20260928 14:55:54.410243 2200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10803799cfee44bcb6e9dca381e52103. Latest consensus state: current_term: 1 leader_uuid: "10803799cfee44bcb6e9dca381e52103" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10803799cfee44bcb6e9dca381e52103" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 33181 } } }
I20260928 14:55:54.410233 2201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10803799cfee44bcb6e9dca381e52103" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10803799cfee44bcb6e9dca381e52103" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 33181 } } }
I20260928 14:55:54.410430 2201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103 [sys.catalog]: This master's current role is: LEADER
I20260928 14:55:54.410940 2203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:55:54.410364 2200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103 [sys.catalog]: This master's current role is: LEADER
I20260928 14:55:54.411527 2203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:55:54.412616 2197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:54.400570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47892 (local address 127.20.116.190:33181)
0928 14:55:54.400786 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:54.400796 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:54.400843 (+ 47us) server_negotiation.cc:407] Connection header received
0928 14:55:54.401322 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:54.401331 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:54.401405 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:54.401562 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:54.402179 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:54.403621 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:54.404452 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:54.404917 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:54.405052 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:54.407848 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:54.407852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:54.407852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:54.407865 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:54.410340 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:54.410811 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:54.410815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:54.410815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:54.410923 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:54.411163 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:54.411164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:54.411164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:54.411498 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:54.411662 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:54.412204 (+ 542us) server_negotiation.cc:299] Negotiation successful
0928 14:55:54.412377 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":101,"threads_started":1}
I20260928 14:55:54.413699 2203 catalog_manager.cc:1347] Generated new cluster ID: 9dd2c9abb76d4ddd873e252332e7e14e
I20260928 14:55:54.413769 2203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:55:54.413795 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:55:54.433343 2203 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:55:54.434247 2203 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:55:54.439680 2203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10803799cfee44bcb6e9dca381e52103: Generated new TSK 0
I20260928 14:55:54.439925 2203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:55:54.467891 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 2126
2026-09-28T14:55:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1369 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:55:54 dist-test-slave-pxl6 krb5kdc[2221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:55:54 dist-test-slave-pxl6 krb5kdc[2221](info): set up 2 sockets
Sep 28 14:55:54 dist-test-slave-pxl6 krb5kdc[2221](info): commencing operation
krb5kdc: starting...
W20260928 14:55:55.523659 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:55:55 dist-test-slave-pxl6 krb5kdc[2221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:55:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:55:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:55:55.650156 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:43661
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:43661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:55:55.736393 2234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:55.736696 2234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:55.736733 2234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:55.738412 2234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:55:55.738472 2234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:55.738487 2234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:55.738500 2234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:55:55.738513 2234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:55:55.740154 2234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:43661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:43661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:55.740437 2234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:55.740684 2234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:55.745868 2242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:55.745923 2234 server_base.cc:1034] running on GCE node
W20260928 14:55:55.745874 2240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:55.745891 2239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:55.746474 2234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:55.746853 2234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:55.748023 2234 hybrid_clock.cc:648] HybridClock initialized: now 1790607355747997 us; error 51 us; skew 500 ppm
Sep 28 14:55:55 dist-test-slave-pxl6 krb5kdc[2221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607355, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:55.750734 2234 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:55:55.751235 2234 webserver.cc:460] Webserver started at http://127.20.116.190:38731/ using document root <none> and password file <none>
I20260928 14:55:55.751487 2234 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:55.751559 2234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:55:55.751706 2234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:55:55.752789 2234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "10ee404f27ab49e1b46ca6daa3e49316"
format_stamp: "Formatted at 2026-09-28 14:55:55 on dist-test-slave-pxl6"
server_key: "e9f446703b8b700ec4cf9f0f9990709a"
server_key_iv: "f22354d35db10724207c71d302392f6f"
server_key_version: "encryptionkey@0"
I20260928 14:55:55.753176 2234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "10ee404f27ab49e1b46ca6daa3e49316"
format_stamp: "Formatted at 2026-09-28 14:55:55 on dist-test-slave-pxl6"
server_key: "e9f446703b8b700ec4cf9f0f9990709a"
server_key_iv: "f22354d35db10724207c71d302392f6f"
server_key_version: "encryptionkey@0"
I20260928 14:55:55.754994 2234 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260928 14:55:55.755949 2249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:55.756142 2234 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:55:55.756275 2234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "10ee404f27ab49e1b46ca6daa3e49316"
format_stamp: "Formatted at 2026-09-28 14:55:55 on dist-test-slave-pxl6"
server_key: "e9f446703b8b700ec4cf9f0f9990709a"
server_key_iv: "f22354d35db10724207c71d302392f6f"
server_key_version: "encryptionkey@0"
I20260928 14:55:55.756350 2234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:55.777733 2234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:55.779891 2234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:55.780032 2234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:55.784287 2234 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:43661
I20260928 14:55:55.784391 2301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:43661 every 8 connection(s)
I20260928 14:55:55.784706 2234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:55:55.785681 2302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:55:55.787088 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 2234
I20260928 14:55:55.787196 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:55:55.787403 20946 external_mini_cluster.cc:1428] Setting key c3de6c5a11a15a24eee5b525b3ba5ab0
I20260928 14:55:55.788584 2302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316: Bootstrap starting.
I20260928 14:55:55.789191 2302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316: Neither blocks nor log segments found. Creating new log.
I20260928 14:55:55.789485 2302 log.cc:824] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316: Log is configured to *not* fsync() on all Append() calls
I20260928 14:55:55.790159 2302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316: No bootstrap required, opened a new log
I20260928 14:55:55.791993 2302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ee404f27ab49e1b46ca6daa3e49316" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43661 } }
I20260928 14:55:55.792156 2302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:55:55.792208 2302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee404f27ab49e1b46ca6daa3e49316, State: Initialized, Role: FOLLOWER
I20260928 14:55:55.792361 2302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ee404f27ab49e1b46ca6daa3e49316" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43661 } }
I20260928 14:55:55.792440 2302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:55:55.792462 2302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:55:55.792527 2302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:55:55.793104 2302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ee404f27ab49e1b46ca6daa3e49316" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43661 } }
I20260928 14:55:55.793195 2302 leader_election.cc:302] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10ee404f27ab49e1b46ca6daa3e49316; no voters:
I20260928 14:55:55.793401 2302 leader_election.cc:288] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:55:55.793428 2307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:55:55.793534 2307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 1 LEADER]: Becoming Leader. State: Replica: 10ee404f27ab49e1b46ca6daa3e49316, State: Running, Role: LEADER
I20260928 14:55:55.793686 2302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:55:55.793684 2307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ee404f27ab49e1b46ca6daa3e49316" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43661 } }
Sep 28 14:55:55 dist-test-slave-pxl6 krb5kdc[2221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:55:55.794190 2309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10ee404f27ab49e1b46ca6daa3e49316. Latest consensus state: current_term: 1 leader_uuid: "10ee404f27ab49e1b46ca6daa3e49316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ee404f27ab49e1b46ca6daa3e49316" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43661 } } }
I20260928 14:55:55.794293 2309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [sys.catalog]: This master's current role is: LEADER
I20260928 14:55:55.794498 2308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10ee404f27ab49e1b46ca6daa3e49316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ee404f27ab49e1b46ca6daa3e49316" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43661 } } }
I20260928 14:55:55.794644 2308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [sys.catalog]: This master's current role is: LEADER
I20260928 14:55:55.794836 2315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:55:55.795778 2315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:55:55.797690 2315 catalog_manager.cc:1347] Generated new cluster ID: 7bd2333a531945a78182e96e40f1025c
I20260928 14:55:55.798357 2315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:55:55.798493 2305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:55.787929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35828 (local address 127.20.116.190:43661)
0928 14:55:55.788111 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:55.788119 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:55.788143 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:55:55.788559 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:55.788565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:55.788611 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:55.788799 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:55.789468 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:55.790457 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:55.791246 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:55.791651 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:55.791775 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:55.794438 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:55.794442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:55.794442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:55.794455 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:55.796547 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:55.797004 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:55.797012 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:55.797013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:55.797126 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:55.797364 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:55.797365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:55.797365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:55.797616 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:55.797827 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:55.798079 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 14:55:55.798284 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260928 14:55:55.813728 2315 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:55:55.814504 2315 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:55:55.823935 2315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316: Generated new TSK 0
I20260928 14:55:55.824168 2315 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
I20260928 14:55:55.861362 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:43661
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:55:55.946683 2328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:55.946899 2328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:55.946924 2328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:55.948125 2328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:55.948171 2328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:55.948201 2328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:55:55.949702 2328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:43661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:55.949965 2328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:55.950212 2328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:55.955487 2336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:55.955504 2334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:55.955515 2333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:55.955660 2328 server_base.cc:1034] running on GCE node
I20260928 14:55:55.955974 2328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:55.956234 2328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:55.957454 2328 hybrid_clock.cc:648] HybridClock initialized: now 1790607355957436 us; error 35 us; skew 500 ppm
Sep 28 14:55:55 dist-test-slave-pxl6 krb5kdc[2221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607355, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:55.960484 2328 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:55:55.961014 2328 webserver.cc:460] Webserver started at http://127.20.116.129:37947/ using document root <none> and password file <none>
I20260928 14:55:55.961293 2328 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:55.961375 2328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:55:55.961541 2328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:55:55.962796 2328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "e636a58f85d14261b150d45a4f21bef4"
format_stamp: "Formatted at 2026-09-28 14:55:55 on dist-test-slave-pxl6"
server_key: "1daf754ae395c00bd4f6a9f89c2e1c38"
server_key_iv: "2f01511f6b810bbefc038da52d574898"
server_key_version: "encryptionkey@0"
I20260928 14:55:55.963198 2328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "e636a58f85d14261b150d45a4f21bef4"
format_stamp: "Formatted at 2026-09-28 14:55:55 on dist-test-slave-pxl6"
server_key: "1daf754ae395c00bd4f6a9f89c2e1c38"
server_key_iv: "2f01511f6b810bbefc038da52d574898"
server_key_version: "encryptionkey@0"
I20260928 14:55:55.964843 2328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:55:55.965719 2343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:55.965957 2328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:55:55.966109 2328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "e636a58f85d14261b150d45a4f21bef4"
format_stamp: "Formatted at 2026-09-28 14:55:55 on dist-test-slave-pxl6"
server_key: "1daf754ae395c00bd4f6a9f89c2e1c38"
server_key_iv: "2f01511f6b810bbefc038da52d574898"
server_key_version: "encryptionkey@0"
I20260928 14:55:55.966199 2328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:55.987946 2328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:55.990319 2328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:55.990520 2328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:55.990813 2328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:55:55.991153 2328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:55:55.991209 2328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:55.991266 2328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:55:55.991287 2328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:55.997522 2328 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:40339
I20260928 14:55:55.997654 2457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:40339 every 8 connection(s)
I20260928 14:55:55.997962 2328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:55:56 dist-test-slave-pxl6 krb5kdc[2221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607355, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:55:56.007086 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 2328
I20260928 14:55:56.007208 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:55:56.007452 20946 external_mini_cluster.cc:1428] Setting key 37855f60c9bfea21fedc83d2b6043612
I20260928 14:55:56.008477 2305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:55.998878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:54237 (local address 127.20.116.190:43661)
0928 14:55:55.998961 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:55.998965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:55.999224 (+ 259us) server_negotiation.cc:407] Connection header received
0928 14:55:56.000208 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:56.000210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:56.000221 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:56.000283 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:56.000977 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.001497 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.002511 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.002771 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.002835 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:56.005101 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:56.005103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:56.005103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:56.005112 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:56.006659 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:56.007257 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:56.007258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:56.007259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:56.007326 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:56.007628 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:56.007629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:56.007629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:56.007796 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:56.007885 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:56.008225 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 14:55:56.008340 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:55:56.008569 2460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:55.998871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.129:54237)
0928 14:55:55.999080 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:55.999120 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:56.000082 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:56.000373 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:56.000376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:56.000636 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:56.000859 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:56.000863 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.001584 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.001585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:56.002413 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:56.002415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.002853 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.002854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:56.002971 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:56.003467 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:56.003475 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:56.004990 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:56.006763 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:56.006772 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:56.006775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:56.007144 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:56.007402 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:56.007403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:56.007403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:56.007553 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:56.007876 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:56.007884 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:56.008037 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 14:55:56.008196 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 14:55:56.008365 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":129,"threads_started":1}
I20260928 14:55:56.009168 2458 heartbeater.cc:347] Connected to a master server at 127.20.116.190:43661
I20260928 14:55:56.009282 2458 heartbeater.cc:464] Registering TS with master...
I20260928 14:55:56.009490 2458 heartbeater.cc:511] Master 127.20.116.190:43661 requested a full tablet report, sending...
I20260928 14:55:56.010105 2266 ts_manager.cc:194] Registered new tserver with Master: e636a58f85d14261b150d45a4f21bef4 (127.20.116.129:40339)
I20260928 14:55:56.011091 2266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:54237
WARNING: no policy specified for kudu/127.20.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
I20260928 14:55:56.037042 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:43661
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:55:56.125492 2463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:56.125722 2463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:56.125749 2463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:56.127022 2463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:56.127066 2463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:56.127094 2463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:55:56.128602 2463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:43661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:56.128881 2463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:56.129122 2463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:56.134420 2471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:56.134404 2468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:56.134404 2469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:56.134505 2463 server_base.cc:1034] running on GCE node
I20260928 14:55:56.134914 2463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:56.135149 2463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:56.136322 2463 hybrid_clock.cc:648] HybridClock initialized: now 1790607356136293 us; error 45 us; skew 500 ppm
Sep 28 14:55:56 dist-test-slave-pxl6 krb5kdc[2221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607356, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:56.139416 2463 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:55:56.139878 2463 webserver.cc:460] Webserver started at http://127.20.116.130:45231/ using document root <none> and password file <none>
I20260928 14:55:56.140075 2463 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:56.140120 2463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:55:56.140211 2463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:55:56.141244 2463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "f0fc0dcbac8f49c19b2e2e12098b287a"
format_stamp: "Formatted at 2026-09-28 14:55:56 on dist-test-slave-pxl6"
server_key: "903e1aa1fe5308626bb27cf16cff9382"
server_key_iv: "96379b06d3136d481fe56e9c35c44c0f"
server_key_version: "encryptionkey@0"
I20260928 14:55:56.141541 2463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "f0fc0dcbac8f49c19b2e2e12098b287a"
format_stamp: "Formatted at 2026-09-28 14:55:56 on dist-test-slave-pxl6"
server_key: "903e1aa1fe5308626bb27cf16cff9382"
server_key_iv: "96379b06d3136d481fe56e9c35c44c0f"
server_key_version: "encryptionkey@0"
I20260928 14:55:56.143030 2463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:55:56.143853 2478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:56.144044 2463 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:55:56.144142 2463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "f0fc0dcbac8f49c19b2e2e12098b287a"
format_stamp: "Formatted at 2026-09-28 14:55:56 on dist-test-slave-pxl6"
server_key: "903e1aa1fe5308626bb27cf16cff9382"
server_key_iv: "96379b06d3136d481fe56e9c35c44c0f"
server_key_version: "encryptionkey@0"
I20260928 14:55:56.144220 2463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:56.154430 2463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:56.157235 2463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:56.157399 2463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:56.157655 2463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:55:56.157971 2463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:55:56.158005 2463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:56.158028 2463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:55:56.158075 2463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:56.164162 2463 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:37939
I20260928 14:55:56.164254 2591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:37939 every 8 connection(s)
I20260928 14:55:56.164614 2463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:55:56 dist-test-slave-pxl6 krb5kdc[2221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607356, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:55:56.172701 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 2463
I20260928 14:55:56.172824 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:55:56.173014 20946 external_mini_cluster.cc:1428] Setting key ba14308bd4792248419856db46d5b9a8
I20260928 14:55:56.174511 2305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:56.165369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:51481 (local address 127.20.116.190:43661)
0928 14:55:56.165415 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:56.165417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:56.165633 (+ 216us) server_negotiation.cc:407] Connection header received
0928 14:55:56.166146 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:56.166147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:56.166156 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:56.166215 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:56.167214 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.167769 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.168789 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.169056 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.169123 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:56.171424 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:56.171426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:56.171426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:56.171432 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:56.172973 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:56.173425 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:56.173427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:56.173427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:56.173491 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:56.173792 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:56.173793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:56.173793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:56.173918 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:56.173973 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:56.174365 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 14:55:56.174442 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:55:56.174791 2594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:56.165327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.130:51481)
0928 14:55:56.165494 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:56.165536 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:56.166055 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:56.166271 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:56.166274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:56.166922 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:56.167118 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:56.167123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.167858 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.167859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:56.168692 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:56.168694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.169138 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.169139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:56.169248 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:56.169872 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:56.169880 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:56.171338 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:56.173064 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:56.173072 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:56.173074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:56.173341 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:56.173557 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:56.173558 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:56.173558 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:56.173713 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:56.173987 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:56.173994 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:56.174185 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 14:55:56.174308 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 14:55:56.174493 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260928 14:55:56.175384 2592 heartbeater.cc:347] Connected to a master server at 127.20.116.190:43661
I20260928 14:55:56.175495 2592 heartbeater.cc:464] Registering TS with master...
I20260928 14:55:56.175673 2592 heartbeater.cc:511] Master 127.20.116.190:43661 requested a full tablet report, sending...
I20260928 14:55:56.176070 2266 ts_manager.cc:194] Registered new tserver with Master: f0fc0dcbac8f49c19b2e2e12098b287a (127.20.116.130:37939)
I20260928 14:55:56.176453 2266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:51481
WARNING: no policy specified for kudu/127.20.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
I20260928 14:55:56.202643 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:43661
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:55:56.286207 2597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:56.286428 2597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:56.286453 2597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:56.287730 2597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:56.287773 2597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:56.287801 2597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:55:56.289206 2597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:43661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:56.289394 2597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:56.289572 2597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:56.294764 2605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:56.294801 2603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:56.294771 2602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:56.294821 2597 server_base.cc:1034] running on GCE node
I20260928 14:55:56.295387 2597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:56.295676 2597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:56.296834 2597 hybrid_clock.cc:648] HybridClock initialized: now 1790607356296816 us; error 35 us; skew 500 ppm
Sep 28 14:55:56 dist-test-slave-pxl6 krb5kdc[2221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607356, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:56.304989 2597 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:55:56.305408 2597 webserver.cc:460] Webserver started at http://127.20.116.131:34959/ using document root <none> and password file <none>
I20260928 14:55:56.305696 2597 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:56.305775 2597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:55:56.305930 2597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:55:56.307124 2597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "aef01926632b40e2a3329141e35bd3d7"
format_stamp: "Formatted at 2026-09-28 14:55:56 on dist-test-slave-pxl6"
server_key: "868414ef3772392aa08e33cb95d350fd"
server_key_iv: "f9208c9698ff82ad140eddd593b1590b"
server_key_version: "encryptionkey@0"
I20260928 14:55:56.307502 2597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "aef01926632b40e2a3329141e35bd3d7"
format_stamp: "Formatted at 2026-09-28 14:55:56 on dist-test-slave-pxl6"
server_key: "868414ef3772392aa08e33cb95d350fd"
server_key_iv: "f9208c9698ff82ad140eddd593b1590b"
server_key_version: "encryptionkey@0"
I20260928 14:55:56.308938 2597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:55:56.309818 2612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:56.310050 2597 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:55:56.310166 2597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "aef01926632b40e2a3329141e35bd3d7"
format_stamp: "Formatted at 2026-09-28 14:55:56 on dist-test-slave-pxl6"
server_key: "868414ef3772392aa08e33cb95d350fd"
server_key_iv: "f9208c9698ff82ad140eddd593b1590b"
server_key_version: "encryptionkey@0"
I20260928 14:55:56.310256 2597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:56.323225 2597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:56.326021 2597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:56.326216 2597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:56.326479 2597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:55:56.326865 2597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:55:56.326902 2597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:56.326926 2597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:55:56.326970 2597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:56.333163 2597 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:36955
I20260928 14:55:56.333237 2725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:36955 every 8 connection(s)
I20260928 14:55:56.333585 2597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:55:56.338052 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 2597
I20260928 14:55:56.338150 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:55:56.338331 20946 external_mini_cluster.cc:1428] Setting key acae3ec51d5813008aa419e1bff97ad7
Sep 28 14:55:56 dist-test-slave-pxl6 krb5kdc[2221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607356, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:55:56.343338 2305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:56.334374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:56551 (local address 127.20.116.190:43661)
0928 14:55:56.334439 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:56.334442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:56.334746 (+ 304us) server_negotiation.cc:407] Connection header received
0928 14:55:56.335229 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:56.335230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:56.335241 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:56.335306 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:56.336240 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.336768 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.337894 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.338209 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.338287 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:56.340747 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:56.340749 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:56.340749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:56.340756 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:56.341963 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:56.342368 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:56.342368 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:56.342369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:56.342420 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:56.342730 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:56.342731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:56.342731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:56.342848 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:56.342905 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:56.343210 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 14:55:56.343273 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 14:55:56.343565 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:56.334363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.131:56551)
0928 14:55:56.334550 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:56.334594 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:56.335139 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:56.335363 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:56.335366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:56.335954 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:56.336160 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:56.336166 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.336884 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.336886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:56.337787 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:56.337790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.338426 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.338427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:56.338607 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:56.339241 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:56.339250 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:56.340659 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:56.342046 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:56.342052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:56.342054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:56.342295 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:56.342479 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:56.342480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:56.342480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:56.342662 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:56.342934 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:56.342941 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:56.343067 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 14:55:56.343164 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 14:55:56.343293 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260928 14:55:56.344130 2726 heartbeater.cc:347] Connected to a master server at 127.20.116.190:43661
I20260928 14:55:56.344244 2726 heartbeater.cc:464] Registering TS with master...
I20260928 14:55:56.344429 2726 heartbeater.cc:511] Master 127.20.116.190:43661 requested a full tablet report, sending...
I20260928 14:55:56.344854 2266 ts_manager.cc:194] Registered new tserver with Master: aef01926632b40e2a3329141e35bd3d7 (127.20.116.131:36955)
I20260928 14:55:56.345230 2266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:56551
I20260928 14:55:56.350296 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:55:56.357945 2305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:56.351772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35832 (local address 127.20.116.190:43661)
0928 14:55:56.351836 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:56.351838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:56.352116 (+ 278us) server_negotiation.cc:407] Connection header received
0928 14:55:56.352143 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:56.352144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:56.352155 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:56.352220 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:56.352786 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.353263 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.353997 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.354180 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.354228 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:56.355502 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:56.355503 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:56.355504 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:56.355510 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:56.356782 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:56.357093 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:56.357094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:56.357094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:56.357144 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:56.357357 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:56.357358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:56.357358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:56.357555 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:56.357637 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:56.357814 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 14:55:56.357878 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:55:56.359716 2266 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:55:56.361465 2266 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:55:56.368580 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:56.364562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:36955 (local address 127.0.0.1:56788)
0928 14:55:56.365048 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:56.365074 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:56.365128 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:56.365430 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:56.365430 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:56.365432 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:55:56.365468 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:56.365470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.366560 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.366561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:56.367332 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:56.367334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.368217 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.368218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:56.368299 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:56.368314 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 14:55:56.368438 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 14:55:56.368502 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":58,"threads_started":1}
I20260928 14:55:56.368597 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:56.364739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56788 (local address 127.20.116.131:36955)
0928 14:55:56.365265 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:56.365269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:56.365278 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:55:56.365320 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:56.365322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:56.365337 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:56.365418 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:55:56.365526 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.366435 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.367426 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.368126 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.368207 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:56.368264 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:56.368465 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 14:55:56.368506 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":90,"threads_started":1}
I20260928 14:55:56.368872 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:56.364157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:37939 (local address 127.0.0.1:37758)
0928 14:55:56.364279 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:56.364313 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:56.364395 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:56.364734 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:56.364737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:56.364740 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:55:56.364815 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:56.364818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.366190 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.366192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:56.367739 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:56.367741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.368623 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.368624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:56.368687 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:56.368692 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:55:56.368748 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 14:55:56.368796 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":24,"thread_start_us":78,"threads_started":1}
I20260928 14:55:56.369148 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:56.364774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59486 (local address 127.20.116.129:40339)
0928 14:55:56.364982 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:56.364987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:56.364997 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:55:56.365035 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:56.365037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:56.365051 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:56.365132 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:55:56.365275 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.366083 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.367652 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.368370 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.368471 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:56.368524 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:56.369008 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 14:55:56.369057 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
I20260928 14:55:56.369210 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:56.364267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37758 (local address 127.20.116.130:37939)
0928 14:55:56.364464 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:56.364470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:56.364481 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:55:56.364523 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:56.364530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:56.364543 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:56.364636 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:55:56.364907 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.366072 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.367825 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.368476 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.368538 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:56.368589 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:56.369024 (+ 435us) server_negotiation.cc:299] Negotiation successful
0928 14:55:56.369116 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":99,"threads_started":1}
I20260928 14:55:56.369527 2660 tablet_service.cc:1467] Processing CreateTablet for tablet 11e12c70203746558a5c2b70860f73cc (DEFAULT_TABLE table=test-table [id=dd0632eca481415bb6bc33552c2818a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:55:56.369917 2660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e12c70203746558a5c2b70860f73cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:55:56.369935 2526 tablet_service.cc:1467] Processing CreateTablet for tablet 11e12c70203746558a5c2b70860f73cc (DEFAULT_TABLE table=test-table [id=dd0632eca481415bb6bc33552c2818a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:55:56.370307 2526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e12c70203746558a5c2b70860f73cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:55:56.370728 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:56.364386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:40339 (local address 127.0.0.1:59486)
0928 14:55:56.364508 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:56.364518 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:56.364564 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:56.365132 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:56.365133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:56.365135 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:55:56.365189 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:56.365190 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.366219 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.366220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:56.367566 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:56.367569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.368759 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.368761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:56.368826 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:56.368830 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:55:56.370592 (+ 1762us) client_negotiation.cc:240] Negotiation successful
0928 14:55:56.370658 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":50,"thread_start_us":81,"threads_started":1}
I20260928 14:55:56.372892 2743 tablet_bootstrap.cc:492] T 11e12c70203746558a5c2b70860f73cc P aef01926632b40e2a3329141e35bd3d7: Bootstrap starting.
I20260928 14:55:56.372942 2744 tablet_bootstrap.cc:492] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a: Bootstrap starting.
I20260928 14:55:56.373651 2744 tablet_bootstrap.cc:654] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a: Neither blocks nor log segments found. Creating new log.
I20260928 14:55:56.373831 2743 tablet_bootstrap.cc:654] T 11e12c70203746558a5c2b70860f73cc P aef01926632b40e2a3329141e35bd3d7: Neither blocks nor log segments found. Creating new log.
I20260928 14:55:56.373904 2392 tablet_service.cc:1467] Processing CreateTablet for tablet 11e12c70203746558a5c2b70860f73cc (DEFAULT_TABLE table=test-table [id=dd0632eca481415bb6bc33552c2818a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:55:56.374256 2744 log.cc:824] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:55:56.374289 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e12c70203746558a5c2b70860f73cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:55:56.374415 2743 log.cc:824] T 11e12c70203746558a5c2b70860f73cc P aef01926632b40e2a3329141e35bd3d7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:55:56.375092 2744 tablet_bootstrap.cc:492] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a: No bootstrap required, opened a new log
I20260928 14:55:56.375204 2744 ts_tablet_manager.cc:1397] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:55:56.375277 2743 tablet_bootstrap.cc:492] T 11e12c70203746558a5c2b70860f73cc P aef01926632b40e2a3329141e35bd3d7: No bootstrap required, opened a new log
I20260928 14:55:56.375348 2743 ts_tablet_manager.cc:1397] T 11e12c70203746558a5c2b70860f73cc P aef01926632b40e2a3329141e35bd3d7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:55:56.376792 2744 raft_consensus.cc:348] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } }
I20260928 14:55:56.376981 2744 raft_consensus.cc:374] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:55:56.377020 2744 raft_consensus.cc:729] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0fc0dcbac8f49c19b2e2e12098b287a, State: Initialized, Role: FOLLOWER
I20260928 14:55:56.377162 2744 consensus_queue.cc:260] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } }
I20260928 14:55:56.377465 2592 heartbeater.cc:503] Master 127.20.116.190:43661 was elected leader, sending a full tablet report...
I20260928 14:55:56.377417 2743 raft_consensus.cc:348] T 11e12c70203746558a5c2b70860f73cc P aef01926632b40e2a3329141e35bd3d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } }
I20260928 14:55:56.377624 2743 raft_consensus.cc:374] T 11e12c70203746558a5c2b70860f73cc P aef01926632b40e2a3329141e35bd3d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:55:56.377476 2744 ts_tablet_manager.cc:1428] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 14:55:56.377718 2743 raft_consensus.cc:729] T 11e12c70203746558a5c2b70860f73cc P aef01926632b40e2a3329141e35bd3d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aef01926632b40e2a3329141e35bd3d7, State: Initialized, Role: FOLLOWER
I20260928 14:55:56.377853 2743 consensus_queue.cc:260] T 11e12c70203746558a5c2b70860f73cc P aef01926632b40e2a3329141e35bd3d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } }
I20260928 14:55:56.378140 2743 ts_tablet_manager.cc:1428] T 11e12c70203746558a5c2b70860f73cc P aef01926632b40e2a3329141e35bd3d7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:55:56.378186 2726 heartbeater.cc:503] Master 127.20.116.190:43661 was elected leader, sending a full tablet report...
I20260928 14:55:56.378324 2748 tablet_bootstrap.cc:492] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4: Bootstrap starting.
I20260928 14:55:56.379081 2748 tablet_bootstrap.cc:654] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4: Neither blocks nor log segments found. Creating new log.
I20260928 14:55:56.379616 2748 log.cc:824] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:55:56.380338 2748 tablet_bootstrap.cc:492] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4: No bootstrap required, opened a new log
I20260928 14:55:56.380396 2748 ts_tablet_manager.cc:1397] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:55:56.382020 2748 raft_consensus.cc:348] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } }
I20260928 14:55:56.382172 2748 raft_consensus.cc:374] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:55:56.382195 2748 raft_consensus.cc:729] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e636a58f85d14261b150d45a4f21bef4, State: Initialized, Role: FOLLOWER
I20260928 14:55:56.382320 2748 consensus_queue.cc:260] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } }
I20260928 14:55:56.382664 2748 ts_tablet_manager.cc:1428] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:55:56.382688 2458 heartbeater.cc:503] Master 127.20.116.190:43661 was elected leader, sending a full tablet report...
W20260928 14:55:56.415238 2593 tablet.cc:2369] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:55:56.498764 2459 tablet.cc:2369] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:55:56.584234 2727 tablet.cc:2369] T 11e12c70203746558a5c2b70860f73cc P aef01926632b40e2a3329141e35bd3d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:55:56.624778 2747 raft_consensus.cc:482] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:55:56.624912 2747 raft_consensus.cc:504] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } }
I20260928 14:55:56.625413 2747 leader_election.cc:288] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e636a58f85d14261b150d45a4f21bef4 (127.20.116.129:40339), aef01926632b40e2a3329141e35bd3d7 (127.20.116.131:36955)
I20260928 14:55:56.628649 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:56.625947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:39827 (local address 127.20.116.131:36955)
0928 14:55:56.625999 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:56.626002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:56.626094 (+ 92us) server_negotiation.cc:407] Connection header received
0928 14:55:56.626143 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:56.626145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:56.626175 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:56.626231 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:55:56.626397 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.626924 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.627877 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.628260 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.628348 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:56.628358 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:56.628525 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 14:55:56.628571 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 14:55:56.628675 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:56.625769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:50179 (local address 127.20.116.129:40339)
0928 14:55:56.625847 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:56.625850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:56.625863 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:55:56.625898 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:56.625899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:56.625913 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:56.625990 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:55:56.626249 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.626873 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.627887 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.628261 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.628348 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:56.628358 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:56.628570 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 14:55:56.628614 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:55:56.628675 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:56.625828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:36955 (local address 127.20.116.130:39827)
0928 14:55:56.626017 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:56.626028 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:56.626085 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:56.626250 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:56.626251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:56.626252 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:55:56.626328 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:56.626330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.627020 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.627021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:56.627775 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:56.627777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.628348 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.628349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:56.628447 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:56.628451 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:55:56.628494 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 14:55:56.628550 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":2,"thread_start_us":129,"threads_started":1}
I20260928 14:55:56.628683 2594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:56.625690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:40339 (local address 127.20.116.130:50179)
0928 14:55:56.625756 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:56.625769 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:56.625819 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:56.626017 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:56.626018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:56.626020 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:55:56.626121 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:56.626124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.626954 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.626956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:56.627782 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:56.627785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.628347 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.628348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:56.628467 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:56.628470 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:55:56.628542 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 14:55:56.628580 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":97}
I20260928 14:55:56.629066 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e12c70203746558a5c2b70860f73cc" candidate_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aef01926632b40e2a3329141e35bd3d7" is_pre_election: true
I20260928 14:55:56.629257 2680 raft_consensus.cc:2454] T 11e12c70203746558a5c2b70860f73cc P aef01926632b40e2a3329141e35bd3d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0fc0dcbac8f49c19b2e2e12098b287a in term 0.
I20260928 14:55:56.629233 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e12c70203746558a5c2b70860f73cc" candidate_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e636a58f85d14261b150d45a4f21bef4" is_pre_election: true
I20260928 14:55:56.629392 2412 raft_consensus.cc:2454] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0fc0dcbac8f49c19b2e2e12098b287a in term 0.
I20260928 14:55:56.629550 2482 leader_election.cc:302] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aef01926632b40e2a3329141e35bd3d7, f0fc0dcbac8f49c19b2e2e12098b287a; no voters:
I20260928 14:55:56.629699 2747 raft_consensus.cc:2781] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:55:56.629760 2747 raft_consensus.cc:482] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:55:56.629787 2747 raft_consensus.cc:3037] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:55:56.630515 2747 raft_consensus.cc:504] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } }
I20260928 14:55:56.630719 2747 leader_election.cc:288] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a [CANDIDATE]: Term 1 election: Requested vote from peers e636a58f85d14261b150d45a4f21bef4 (127.20.116.129:40339), aef01926632b40e2a3329141e35bd3d7 (127.20.116.131:36955)
I20260928 14:55:56.630890 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e12c70203746558a5c2b70860f73cc" candidate_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e636a58f85d14261b150d45a4f21bef4"
I20260928 14:55:56.630924 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e12c70203746558a5c2b70860f73cc" candidate_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aef01926632b40e2a3329141e35bd3d7"
I20260928 14:55:56.630990 2680 raft_consensus.cc:3037] T 11e12c70203746558a5c2b70860f73cc P aef01926632b40e2a3329141e35bd3d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:55:56.630995 2412 raft_consensus.cc:3037] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:55:56.631735 2680 raft_consensus.cc:2454] T 11e12c70203746558a5c2b70860f73cc P aef01926632b40e2a3329141e35bd3d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0fc0dcbac8f49c19b2e2e12098b287a in term 1.
I20260928 14:55:56.631909 2412 raft_consensus.cc:2454] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0fc0dcbac8f49c19b2e2e12098b287a in term 1.
I20260928 14:55:56.631903 2482 leader_election.cc:302] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aef01926632b40e2a3329141e35bd3d7, f0fc0dcbac8f49c19b2e2e12098b287a; no voters:
I20260928 14:55:56.632126 2747 raft_consensus.cc:2781] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:55:56.632295 2747 raft_consensus.cc:686] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a [term 1 LEADER]: Becoming Leader. State: Replica: f0fc0dcbac8f49c19b2e2e12098b287a, State: Running, Role: LEADER
I20260928 14:55:56.632401 2747 consensus_queue.cc:237] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } }
I20260928 14:55:56.633327 2266 catalog_manager.cc:5495] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a reported cstate change: term changed from 0 to 1, leader changed from <none> to f0fc0dcbac8f49c19b2e2e12098b287a (127.20.116.130). New cstate: current_term: 1 leader_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } health_report { overall_health: UNKNOWN } } }
I20260928 14:55:56.653684 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:56.651229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37766 (local address 127.20.116.130:37939)
0928 14:55:56.651293 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:56.651295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:56.651304 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:55:56.651351 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:56.651352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:56.651372 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:56.651431 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:55:56.651537 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.651997 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.652807 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.653009 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.653078 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:56.653315 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:55:56.653425 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:55:56.653491 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:56.653528 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 14:55:56.653549 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:55:56.655855 2680 raft_consensus.cc:1261] T 11e12c70203746558a5c2b70860f73cc P aef01926632b40e2a3329141e35bd3d7 [term 1 FOLLOWER]: Refusing update from remote peer f0fc0dcbac8f49c19b2e2e12098b287a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:55:56.655855 2412 raft_consensus.cc:1261] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4 [term 1 FOLLOWER]: Refusing update from remote peer f0fc0dcbac8f49c19b2e2e12098b287a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:55:56.656270 2753 consensus_queue.cc:1035] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:55:56.656425 2747 consensus_queue.cc:1035] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a [LEADER]: Connected to new peer: Peer: permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:55:56.660156 2757 mvcc.cc:205] Tried to move back new op lower bound from 7334327732858834944 to 7334327732766916608. Current Snapshot: MvccSnapshot[applied={T|T < 7334327732858834944}]
I20260928 14:55:56.661420 2266 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:55:56.661526 2266 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:55:56.662261 2266 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316: Sending DeleteTablet for 3 replicas of tablet 11e12c70203746558a5c2b70860f73cc
I20260928 14:55:56.662616 2526 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e12c70203746558a5c2b70860f73cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:55:56 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:37758
I20260928 14:55:56.662742 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e12c70203746558a5c2b70860f73cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:55:56 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:59486
I20260928 14:55:56.662745 2660 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e12c70203746558a5c2b70860f73cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:55:56 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:56788
I20260928 14:55:56.662988 2768 tablet_replica.cc:330] T 11e12c70203746558a5c2b70860f73cc P aef01926632b40e2a3329141e35bd3d7: stopping tablet replica
I20260928 14:55:56.662993 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 2328
I20260928 14:55:56.663110 2768 raft_consensus.cc:2229] T 11e12c70203746558a5c2b70860f73cc P aef01926632b40e2a3329141e35bd3d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:55:56.663218 2768 raft_consensus.cc:2258] T 11e12c70203746558a5c2b70860f73cc P aef01926632b40e2a3329141e35bd3d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:55:56.663744 2768 ts_tablet_manager.cc:1905] T 11e12c70203746558a5c2b70860f73cc P aef01926632b40e2a3329141e35bd3d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:55:56.665426 2768 ts_tablet_manager.cc:1918] T 11e12c70203746558a5c2b70860f73cc P aef01926632b40e2a3329141e35bd3d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:55:56.665506 2768 log.cc:1196] T 11e12c70203746558a5c2b70860f73cc P aef01926632b40e2a3329141e35bd3d7: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/11e12c70203746558a5c2b70860f73cc
I20260928 14:55:56.665726 2766 tablet_replica.cc:330] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a: stopping tablet replica
I20260928 14:55:56.665799 2768 ts_tablet_manager.cc:1939] T 11e12c70203746558a5c2b70860f73cc P aef01926632b40e2a3329141e35bd3d7: Deleting consensus metadata
I20260928 14:55:56.665823 2766 raft_consensus.cc:2229] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:55:56.665918 2766 raft_consensus.cc:2258] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:55:56.666244 2253 catalog_manager.cc:4841] TS aef01926632b40e2a3329141e35bd3d7 (127.20.116.131:36955): tablet 11e12c70203746558a5c2b70860f73cc (table test-table [id=dd0632eca481415bb6bc33552c2818a8]) successfully deleted
I20260928 14:55:56.666270 2766 ts_tablet_manager.cc:1905] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:55:56.667554 2766 ts_tablet_manager.cc:1918] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:55:56.667645 2766 log.cc:1196] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/11e12c70203746558a5c2b70860f73cc
I20260928 14:55:56.667908 2766 ts_tablet_manager.cc:1939] T 11e12c70203746558a5c2b70860f73cc P f0fc0dcbac8f49c19b2e2e12098b287a: Deleting consensus metadata
I20260928 14:55:56.668280 2253 catalog_manager.cc:4841] TS f0fc0dcbac8f49c19b2e2e12098b287a (127.20.116.130:37939): tablet 11e12c70203746558a5c2b70860f73cc (table test-table [id=dd0632eca481415bb6bc33552c2818a8]) successfully deleted
W20260928 14:55:56.670722 2253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.116.129:40339 (error 108)
I20260928 14:55:56.670838 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 2463
I20260928 14:55:56.672813 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:56.672658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:40339 (local address 127.0.0.1:59498)
0928 14:55:56.672721 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:56.672768 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.129:40339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 14:55:56.672983 2253 catalog_manager.cc:4584] TS e636a58f85d14261b150d45a4f21bef4 (127.20.116.129:40339): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 11e12c70203746558a5c2b70860f73cc: Network error: Client connection negotiation failed: client connection to 127.20.116.129:40339: connect: Connection refused (error 111)
I20260928 14:55:56.676909 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 2597
I20260928 14:55:56.681814 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 2234
I20260928 14:55:56.687440 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:43661
--webserver_interface=127.20.116.190
--webserver_port=38731
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:43661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:55:56.769809 2770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:56.770066 2770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:56.770105 2770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:56.771345 2770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:55:56.771420 2770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:56.771445 2770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:56.771478 2770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:55:56.771507 2770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:55:56.773068 2770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:43661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:43661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=38731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:56.773368 2770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:56.773624 2770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:56.784101 2776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:56.784102 2777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:56.784102 2779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:56.784102 2770 server_base.cc:1034] running on GCE node
I20260928 14:55:56.784627 2770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:56.785000 2770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:56.786173 2770 hybrid_clock.cc:648] HybridClock initialized: now 1790607356786161 us; error 56 us; skew 500 ppm
Sep 28 14:55:56 dist-test-slave-pxl6 krb5kdc[2221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607356, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:56.788805 2770 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:55:56.789227 2770 webserver.cc:460] Webserver started at http://127.20.116.190:38731/ using document root <none> and password file <none>
I20260928 14:55:56.789424 2770 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:56.789475 2770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:55:56.790859 2770 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:55:56.791567 2786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:56.791821 2770 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:55:56.791919 2770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "10ee404f27ab49e1b46ca6daa3e49316"
format_stamp: "Formatted at 2026-09-28 14:55:55 on dist-test-slave-pxl6"
server_key: "e9f446703b8b700ec4cf9f0f9990709a"
server_key_iv: "f22354d35db10724207c71d302392f6f"
server_key_version: "encryptionkey@0"
I20260928 14:55:56.792438 2770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:56.803321 2770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:56.805313 2770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:56.805469 2770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:56.809551 2770 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:43661
I20260928 14:55:56.809639 2838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:43661 every 8 connection(s)
I20260928 14:55:56.810019 2770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:55:56.811712 2839 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:55:56.812382 2839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316: Bootstrap starting.
I20260928 14:55:56.812630 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 2770
I20260928 14:55:56.812939 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:40339
--local_ip_for_outbound_sockets=127.20.116.129
--tserver_master_addrs=127.20.116.190:43661
--webserver_port=37947
--webserver_interface=127.20.116.129
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:55:56.821925 2839 log.cc:824] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316: Log is configured to *not* fsync() on all Append() calls
I20260928 14:55:56.824012 2839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:55:56.824361 2839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316: Bootstrap complete.
I20260928 14:55:56.826836 2839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ee404f27ab49e1b46ca6daa3e49316" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43661 } }
I20260928 14:55:56.827077 2839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee404f27ab49e1b46ca6daa3e49316, State: Initialized, Role: FOLLOWER
I20260928 14:55:56.827272 2839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ee404f27ab49e1b46ca6daa3e49316" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43661 } }
I20260928 14:55:56.827335 2839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:55:56.827401 2839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:55:56.827452 2839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:55:56.828536 2839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ee404f27ab49e1b46ca6daa3e49316" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43661 } }
I20260928 14:55:56.828656 2839 leader_election.cc:302] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10ee404f27ab49e1b46ca6daa3e49316; no voters:
I20260928 14:55:56.828881 2839 leader_election.cc:288] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:55:56.828922 2843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:55:56.829088 2843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 2 LEADER]: Becoming Leader. State: Replica: 10ee404f27ab49e1b46ca6daa3e49316, State: Running, Role: LEADER
I20260928 14:55:56.829201 2839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:55:56.829239 2843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ee404f27ab49e1b46ca6daa3e49316" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43661 } }
I20260928 14:55:56.829602 2844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "10ee404f27ab49e1b46ca6daa3e49316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ee404f27ab49e1b46ca6daa3e49316" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43661 } } }
I20260928 14:55:56.829594 2843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10ee404f27ab49e1b46ca6daa3e49316. Latest consensus state: current_term: 2 leader_uuid: "10ee404f27ab49e1b46ca6daa3e49316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ee404f27ab49e1b46ca6daa3e49316" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43661 } } }
I20260928 14:55:56.829723 2844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [sys.catalog]: This master's current role is: LEADER
I20260928 14:55:56.829728 2843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [sys.catalog]: This master's current role is: LEADER
I20260928 14:55:56.830282 2851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:55:56.831173 2851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:55:56.831277 2851 catalog_manager.cc:1259] Loaded cluster ID: 7bd2333a531945a78182e96e40f1025c
I20260928 14:55:56.831298 2851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:55:56.831518 2851 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 14:55:56.836220 2851 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 14:55:56.838792 2851 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:55:56.839013 2851 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316: Loaded TSK: 0
I20260928 14:55:56.839238 2851 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:55:56.903123 2841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:56.903352 2841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:56.903379 2841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:56.904685 2841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:56.904731 2841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:56.904760 2841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:55:56.906359 2841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:40339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=37947
--tserver_master_addrs=127.20.116.190:43661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:56.906651 2841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:56.906910 2841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:56.912936 2867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:56.912951 2866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:56.913180 2841 server_base.cc:1034] running on GCE node
W20260928 14:55:56.913264 2869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:56.913488 2841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:56.913767 2841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:56.914928 2841 hybrid_clock.cc:648] HybridClock initialized: now 1790607356914909 us; error 32 us; skew 500 ppm
Sep 28 14:55:56 dist-test-slave-pxl6 krb5kdc[2221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607356, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:56.917430 2841 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:55:56.917928 2841 webserver.cc:460] Webserver started at http://127.20.116.129:37947/ using document root <none> and password file <none>
I20260928 14:55:56.918167 2841 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:56.918255 2841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:55:56.919695 2841 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:55:56.920384 2876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:56.920531 2841 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:55:56.920637 2841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "e636a58f85d14261b150d45a4f21bef4"
format_stamp: "Formatted at 2026-09-28 14:55:55 on dist-test-slave-pxl6"
server_key: "1daf754ae395c00bd4f6a9f89c2e1c38"
server_key_iv: "2f01511f6b810bbefc038da52d574898"
server_key_version: "encryptionkey@0"
I20260928 14:55:56.920991 2841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:56.947163 2841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:56.949456 2841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:56.949605 2841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:56.949891 2841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:55:56.950428 2883 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 14:55:56.952112 2841 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:55:56.952164 2841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 14:55:56.952203 2841 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:55:56.952936 2841 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:55:56.952981 2841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 14:55:56.953029 2883 tablet_bootstrap.cc:492] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4: Bootstrap starting.
I20260928 14:55:56.961535 2841 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:40339
I20260928 14:55:56.962121 2841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:55:56.962692 2990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:40339 every 8 connection(s)
I20260928 14:55:56.966545 2883 log.cc:824] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:55:56.968843 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 2841
I20260928 14:55:56.969146 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:37939
--local_ip_for_outbound_sockets=127.20.116.130
--tserver_master_addrs=127.20.116.190:43661
--webserver_port=45231
--webserver_interface=127.20.116.130
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:55:56.969485 2883 tablet_bootstrap.cc:492] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:55:56.969822 2883 tablet_bootstrap.cc:492] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4: Bootstrap complete.
I20260928 14:55:56.971266 2883 ts_tablet_manager.cc:1397] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.009s
Sep 28 14:55:56 dist-test-slave-pxl6 krb5kdc[2221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607356, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:55:56.972658 2883 raft_consensus.cc:348] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } }
I20260928 14:55:56.973021 2883 raft_consensus.cc:729] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e636a58f85d14261b150d45a4f21bef4, State: Initialized, Role: FOLLOWER
I20260928 14:55:56.973244 2883 consensus_queue.cc:260] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } }
I20260928 14:55:56.973598 2883 ts_tablet_manager.cc:1428] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 14:55:56.976413 2993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:56.963004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.129:37157)
0928 14:55:56.963206 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:56.963254 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:56.964147 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:56.964663 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:56.964668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:56.965180 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:56.965532 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:56.965542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.966983 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.966984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:56.968378 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:56.968381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.969209 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.969210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:56.969371 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:56.970021 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:56.970033 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:56.972118 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:56.974481 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:56.974489 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:56.974492 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:56.974790 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:56.975101 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:56.975102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:56.975103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:56.975233 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:56.975674 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:56.975681 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:56.975840 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 14:55:56.975963 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 14:55:56.976185 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":117,"threads_started":1}
I20260928 14:55:56.976413 2994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:56.963093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:37157 (local address 127.20.116.190:43661)
0928 14:55:56.963279 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:56.963288 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:56.963342 (+ 54us) server_negotiation.cc:407] Connection header received
0928 14:55:56.964377 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:56.964383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:56.964427 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:56.964544 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:56.965696 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.966862 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.968481 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.969080 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.969213 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:56.972257 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:56.972262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:56.972263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:56.972276 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:56.974355 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:56.974895 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:56.974899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:56.974899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:56.975014 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:56.975315 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:56.975316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:56.975317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:56.975587 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:56.975697 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:56.976026 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 14:55:56.976186 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":107,"threads_started":1}
I20260928 14:55:56.977156 2991 heartbeater.cc:347] Connected to a master server at 127.20.116.190:43661
I20260928 14:55:56.977247 2991 heartbeater.cc:464] Registering TS with master...
I20260928 14:55:56.977473 2991 heartbeater.cc:511] Master 127.20.116.190:43661 requested a full tablet report, sending...
I20260928 14:55:56.978294 2803 ts_manager.cc:194] Registered new tserver with Master: e636a58f85d14261b150d45a4f21bef4 (127.20.116.129:40339)
I20260928 14:55:56.979578 2803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:37157
I20260928 14:55:56.980747 2991 heartbeater.cc:503] Master 127.20.116.190:43661 was elected leader, sending a full tablet report...
Sep 28 14:55:56 dist-test-slave-pxl6 krb5kdc[2221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607356, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for kudu/127.20.116.129@KRBTEST.COM
I20260928 14:55:56.988468 2998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:56.979046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:40339 (local address 127.0.0.1:59504)
0928 14:55:56.979262 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:56.979305 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:56.979372 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:56.980323 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:56.980329 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:56.980679 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:56.980731 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:56.980733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.981802 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.981803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:56.982499 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:56.982501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.983116 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.983117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:56.983208 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:56.983718 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:56.983722 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:56.985293 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:56.987381 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:56.987387 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:56.987389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:56.987585 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:56.987818 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:56.987819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:56.987819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:56.987874 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:56.988167 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:56.988173 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:56.988227 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 14:55:56.988293 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 14:55:56.988396 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":129,"threads_started":1}
I20260928 14:55:56.988535 2999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:56.979916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59504 (local address 127.20.116.129:40339)
0928 14:55:56.980138 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:56.980143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:56.980155 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:55:56.980191 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:56.980193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:56.980211 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:56.980305 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:56.980820 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.981710 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.982678 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:56.983007 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:56.983112 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:56.985384 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:56.985392 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:56.985393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:56.985401 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:56.987274 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:56.987674 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:56.987677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:56.987678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:56.987745 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:56.987934 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:56.987935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:56.987936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:56.988103 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:56.988184 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:56.988350 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 14:55:56.988434 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":191,"thread_start_us":116,"threads_started":1}
I20260928 14:55:56.988987 2907 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e12c70203746558a5c2b70860f73cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:55:56 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:59504
I20260928 14:55:56.989615 3000 tablet_replica.cc:330] stopping tablet replica
I20260928 14:55:56.989759 3000 raft_consensus.cc:2229] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:55:56.989847 3000 raft_consensus.cc:2258] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:55:56.990200 3000 ts_tablet_manager.cc:1905] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:55:56.992393 3000 ts_tablet_manager.cc:1918] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:55:56.992501 3000 log.cc:1196] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/wal/wals/11e12c70203746558a5c2b70860f73cc
I20260928 14:55:56.992779 3000 ts_tablet_manager.cc:1939] T 11e12c70203746558a5c2b70860f73cc P e636a58f85d14261b150d45a4f21bef4: Deleting consensus metadata
I20260928 14:55:56.993346 2790 catalog_manager.cc:4841] TS e636a58f85d14261b150d45a4f21bef4 (127.20.116.129:40339): tablet 11e12c70203746558a5c2b70860f73cc (table test-table [id=dd0632eca481415bb6bc33552c2818a8]) successfully deleted
W20260928 14:55:57.058809 2996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:57.059159 2996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:57.059191 2996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:57.060549 2996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:57.060603 2996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:57.060634 2996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:55:57.062234 2996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:37939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=45231
--tserver_master_addrs=127.20.116.190:43661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:57.062563 2996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:57.062824 2996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:55:57.068526 2996 server_base.cc:1034] running on GCE node
W20260928 14:55:57.068498 3006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:57.068507 3008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:57.068498 3005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:57.069015 2996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:57.069324 2996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:57.070494 2996 hybrid_clock.cc:648] HybridClock initialized: now 1790607357070472 us; error 37 us; skew 500 ppm
Sep 28 14:55:57 dist-test-slave-pxl6 krb5kdc[2221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607357, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:57.073087 2996 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:55:57.073553 2996 webserver.cc:460] Webserver started at http://127.20.116.130:45231/ using document root <none> and password file <none>
I20260928 14:55:57.073777 2996 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:57.073830 2996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:55:57.075240 2996 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 14:55:57.075825 3015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:57.075992 2996 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:57.076081 2996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "f0fc0dcbac8f49c19b2e2e12098b287a"
format_stamp: "Formatted at 2026-09-28 14:55:56 on dist-test-slave-pxl6"
server_key: "903e1aa1fe5308626bb27cf16cff9382"
server_key_iv: "96379b06d3136d481fe56e9c35c44c0f"
server_key_version: "encryptionkey@0"
I20260928 14:55:57.076436 2996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:57.089956 2996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:57.092226 2996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:57.092386 2996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:57.092674 2996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:55:57.092999 2996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:55:57.093034 2996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:57.093082 2996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:55:57.093110 2996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:57.099099 2996 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:37939
I20260928 14:55:57.099210 3128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:37939 every 8 connection(s)
I20260928 14:55:57.099540 2996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:55:57 dist-test-slave-pxl6 krb5kdc[2221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607357, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:55:57.106245 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 2996
I20260928 14:55:57.106580 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:36955
--local_ip_for_outbound_sockets=127.20.116.131
--tserver_master_addrs=127.20.116.190:43661
--webserver_port=34959
--webserver_interface=127.20.116.131
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:55:57.109480 2994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:57.100363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:52691 (local address 127.20.116.190:43661)
0928 14:55:57.100431 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:57.100437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:57.100680 (+ 243us) server_negotiation.cc:407] Connection header received
0928 14:55:57.101236 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:57.101237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:57.101253 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:57.101308 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:57.102268 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.102849 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.103876 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.104097 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.104173 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:57.106292 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:57.106293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:57.106293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:57.106303 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:57.107874 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:57.108299 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:57.108301 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:57.108301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:57.108364 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:57.108746 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:57.108747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:57.108748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:57.108866 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:57.108932 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:57.109283 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 14:55:57.109386 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:55:57.109689 3131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:57.100363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.130:52691)
0928 14:55:57.100540 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:57.100581 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:57.101126 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:57.101390 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:57.101393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:57.101956 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:57.102161 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:57.102166 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.102947 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.102948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:57.103770 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:57.103773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.104181 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.104183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:57.104312 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:57.104811 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:57.104820 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:57.106185 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:57.107978 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:57.107986 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:57.107988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:57.108222 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:57.108470 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:57.108472 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:57.108472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:57.108640 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:57.108943 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:57.108949 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:57.109136 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 14:55:57.109259 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 14:55:57.109424 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":72,"threads_started":1}
I20260928 14:55:57.110219 3129 heartbeater.cc:347] Connected to a master server at 127.20.116.190:43661
I20260928 14:55:57.110322 3129 heartbeater.cc:464] Registering TS with master...
I20260928 14:55:57.110581 3129 heartbeater.cc:511] Master 127.20.116.190:43661 requested a full tablet report, sending...
I20260928 14:55:57.111066 2803 ts_manager.cc:194] Registered new tserver with Master: f0fc0dcbac8f49c19b2e2e12098b287a (127.20.116.130:37939)
I20260928 14:55:57.111454 2803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:52691
W20260928 14:55:57.190840 3132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:57.191097 3132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:57.191133 3132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:57.192349 3132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:57.192415 3132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:57.192461 3132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:55:57.193981 3132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:36955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=34959
--tserver_master_addrs=127.20.116.190:43661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:57.194270 3132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:57.194550 3132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:57.199893 3137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:57.199905 3138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:57.199903 3140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:57.200114 3132 server_base.cc:1034] running on GCE node
I20260928 14:55:57.200438 3132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:57.200657 3132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:57.201799 3132 hybrid_clock.cc:648] HybridClock initialized: now 1790607357201776 us; error 39 us; skew 500 ppm
Sep 28 14:55:57 dist-test-slave-pxl6 krb5kdc[2221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607357, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:57.212276 3132 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:55:57.212778 3132 webserver.cc:460] Webserver started at http://127.20.116.131:34959/ using document root <none> and password file <none>
I20260928 14:55:57.213029 3132 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:57.213116 3132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:55:57.214646 3132 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:55:57.215353 3147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:57.215565 3132 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:55:57.215672 3132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "aef01926632b40e2a3329141e35bd3d7"
format_stamp: "Formatted at 2026-09-28 14:55:56 on dist-test-slave-pxl6"
server_key: "868414ef3772392aa08e33cb95d350fd"
server_key_iv: "f9208c9698ff82ad140eddd593b1590b"
server_key_version: "encryptionkey@0"
I20260928 14:55:57.216060 3132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:57.245699 3132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:57.247900 3132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:57.248072 3132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:57.248313 3132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:55:57.248623 3132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:55:57.248675 3132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:57.248709 3132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:55:57.248745 3132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:57.255017 3132 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:36955
I20260928 14:55:57.255097 3260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:36955 every 8 connection(s)
I20260928 14:55:57.255465 3132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:55:57 dist-test-slave-pxl6 krb5kdc[2221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607357, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:55:57.262637 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 3132
I20260928 14:55:57.265179 2994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:57.256265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:40745 (local address 127.20.116.190:43661)
0928 14:55:57.256352 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:57.256355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:57.256505 (+ 150us) server_negotiation.cc:407] Connection header received
0928 14:55:57.257116 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:57.257118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:57.257130 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:57.257184 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:57.258161 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.258736 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.259878 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.260082 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.260165 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:57.262145 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:57.262147 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:57.262147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:57.262155 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:57.263722 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:57.264173 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:57.264174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:57.264175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:57.264266 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:57.264531 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:57.264532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:57.264532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:57.264682 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:57.264752 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:57.264997 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 14:55:57.265083 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:55:57.265261 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:57.256193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.131:40745)
0928 14:55:57.256356 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:57.256400 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:57.256991 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:57.257255 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:57.257266 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:57.257846 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:57.258043 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:57.258048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.258830 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.258831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:57.259778 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:57.259781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.260172 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.260173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:57.260299 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:57.260745 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:57.260753 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:57.262043 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:57.263802 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:57.263810 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:57.263812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:57.264094 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:57.264342 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:57.264343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:57.264343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:57.264458 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:57.264743 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:57.264748 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:57.264867 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 14:55:57.264950 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 14:55:57.265064 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260928 14:55:57.265753 3261 heartbeater.cc:347] Connected to a master server at 127.20.116.190:43661
I20260928 14:55:57.265878 3261 heartbeater.cc:464] Registering TS with master...
I20260928 14:55:57.266062 3261 heartbeater.cc:511] Master 127.20.116.190:43661 requested a full tablet report, sending...
I20260928 14:55:57.266486 2803 ts_manager.cc:194] Registered new tserver with Master: aef01926632b40e2a3329141e35bd3d7 (127.20.116.131:36955)
I20260928 14:55:57.266898 2803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:40745
I20260928 14:55:57.270174 3265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:57.263052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35844 (local address 127.20.116.190:43661)
0928 14:55:57.263172 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:57.263174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:57.263187 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:55:57.263267 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:57.263269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:57.263282 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:57.263355 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:57.263888 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.264587 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.265790 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.266067 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.266156 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:57.267431 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:57.267432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:57.267433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:57.267440 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:57.268837 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:57.269270 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:57.269271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:57.269271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:57.269338 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:57.269539 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:57.269540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:57.269540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:57.269709 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:57.269884 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:57.269970 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 14:55:57.270054 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
I20260928 14:55:57.271178 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:55:57.275379 3265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:57.271656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35846 (local address 127.20.116.190:43661)
0928 14:55:57.271727 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:57.271729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:57.271740 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:55:57.271792 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:57.271793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:57.271839 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:57.271909 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:55:57.272081 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.272911 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.274171 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.274675 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.274765 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:57.274992 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:55:57.275135 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:55:57.275240 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:57.275262 (+ 22us) server_negotiation.cc:299] Negotiation successful
0928 14:55:57.275286 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:55:57.277374 2803 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:55:57.280085 2803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:55:57.283015 2907 tablet_service.cc:1467] Processing CreateTablet for tablet 5007b09b832a4c70b24cc897caec918d (DEFAULT_TABLE table=test-table [id=1bc4eaf9f6274a49ad1790e0197856a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:55:57.283237 2907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5007b09b832a4c70b24cc897caec918d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:55:57.285246 2883 tablet_bootstrap.cc:492] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4: Bootstrap starting.
I20260928 14:55:57.285580 2883 tablet_bootstrap.cc:654] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4: Neither blocks nor log segments found. Creating new log.
I20260928 14:55:57.286139 2883 tablet_bootstrap.cc:492] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4: No bootstrap required, opened a new log
I20260928 14:55:57.286195 2883 ts_tablet_manager.cc:1397] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:55:57.286586 2883 raft_consensus.cc:348] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } }
I20260928 14:55:57.286664 2883 raft_consensus.cc:374] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:55:57.286710 2883 raft_consensus.cc:729] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e636a58f85d14261b150d45a4f21bef4, State: Initialized, Role: FOLLOWER
I20260928 14:55:57.286753 2883 consensus_queue.cc:260] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } }
I20260928 14:55:57.286834 2883 ts_tablet_manager.cc:1428] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:55:57.288105 3267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:57.282456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:36955 (local address 127.0.0.1:56790)
0928 14:55:57.282590 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:57.282599 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:57.282639 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:57.283201 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:57.283201 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:57.283203 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:55:57.283244 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:57.283246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.284661 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.284662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:57.286968 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:57.286971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.287839 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.287840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:57.287905 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:57.287931 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 14:55:57.287963 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 14:55:57.288020 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":1184,"thread_start_us":99,"threads_started":1}
I20260928 14:55:57.288226 2998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:57.282361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:37939 (local address 127.0.0.1:37776)
0928 14:55:57.282398 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:57.282408 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:57.282445 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:57.283098 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:57.283099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:57.283101 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:55:57.283157 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:57.283159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.284661 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.284662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:57.285913 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:57.285917 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.287850 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.287851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:57.287906 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:57.287930 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:55:57.287963 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 14:55:57.288020 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":58}
I20260928 14:55:57.288326 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:57.282739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37776 (local address 127.20.116.130:37939)
0928 14:55:57.282892 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:57.282897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:57.282909 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:55:57.282955 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:57.282957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:57.282975 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:57.283339 (+ 364us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:55:57.283415 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.284532 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.286962 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.287730 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.288071 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:57.288149 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:57.288180 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 14:55:57.288236 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260928 14:55:57.288523 3269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:57.282811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56790 (local address 127.20.116.131:36955)
0928 14:55:57.283006 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:57.283010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:57.283018 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:55:57.283055 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:57.283057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:57.283072 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:57.283345 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:55:57.283413 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.284532 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.287067 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.287745 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.288307 (+ 562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:57.288375 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:57.288404 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 14:55:57.288450 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
I20260928 14:55:57.289057 3063 tablet_service.cc:1467] Processing CreateTablet for tablet 5007b09b832a4c70b24cc897caec918d (DEFAULT_TABLE table=test-table [id=1bc4eaf9f6274a49ad1790e0197856a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:55:57.289275 3195 tablet_service.cc:1467] Processing CreateTablet for tablet 5007b09b832a4c70b24cc897caec918d (DEFAULT_TABLE table=test-table [id=1bc4eaf9f6274a49ad1790e0197856a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:55:57.289460 3063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5007b09b832a4c70b24cc897caec918d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:55:57.289615 3195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5007b09b832a4c70b24cc897caec918d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:55:57.293030 3271 tablet_bootstrap.cc:492] T 5007b09b832a4c70b24cc897caec918d P f0fc0dcbac8f49c19b2e2e12098b287a: Bootstrap starting.
I20260928 14:55:57.293687 3272 tablet_bootstrap.cc:492] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7: Bootstrap starting.
I20260928 14:55:57.293984 3271 tablet_bootstrap.cc:654] T 5007b09b832a4c70b24cc897caec918d P f0fc0dcbac8f49c19b2e2e12098b287a: Neither blocks nor log segments found. Creating new log.
I20260928 14:55:57.294453 3272 tablet_bootstrap.cc:654] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7: Neither blocks nor log segments found. Creating new log.
I20260928 14:55:57.294673 3271 log.cc:824] T 5007b09b832a4c70b24cc897caec918d P f0fc0dcbac8f49c19b2e2e12098b287a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:55:57.295078 3272 log.cc:824] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:55:57.295519 3271 tablet_bootstrap.cc:492] T 5007b09b832a4c70b24cc897caec918d P f0fc0dcbac8f49c19b2e2e12098b287a: No bootstrap required, opened a new log
I20260928 14:55:57.295604 3271 ts_tablet_manager.cc:1397] T 5007b09b832a4c70b24cc897caec918d P f0fc0dcbac8f49c19b2e2e12098b287a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:55:57.295799 3272 tablet_bootstrap.cc:492] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7: No bootstrap required, opened a new log
I20260928 14:55:57.295864 3272 ts_tablet_manager.cc:1397] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:55:57.297667 3271 raft_consensus.cc:348] T 5007b09b832a4c70b24cc897caec918d P f0fc0dcbac8f49c19b2e2e12098b287a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } }
I20260928 14:55:57.297850 3271 raft_consensus.cc:374] T 5007b09b832a4c70b24cc897caec918d P f0fc0dcbac8f49c19b2e2e12098b287a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:55:57.297781 3272 raft_consensus.cc:348] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } }
I20260928 14:55:57.297887 3271 raft_consensus.cc:729] T 5007b09b832a4c70b24cc897caec918d P f0fc0dcbac8f49c19b2e2e12098b287a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0fc0dcbac8f49c19b2e2e12098b287a, State: Initialized, Role: FOLLOWER
I20260928 14:55:57.297904 3272 raft_consensus.cc:374] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:55:57.297941 3272 raft_consensus.cc:729] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aef01926632b40e2a3329141e35bd3d7, State: Initialized, Role: FOLLOWER
I20260928 14:55:57.298049 3271 consensus_queue.cc:260] T 5007b09b832a4c70b24cc897caec918d P f0fc0dcbac8f49c19b2e2e12098b287a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } }
I20260928 14:55:57.298049 3272 consensus_queue.cc:260] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } }
I20260928 14:55:57.298352 3272 ts_tablet_manager.cc:1428] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 14:55:57.298352 3271 ts_tablet_manager.cc:1428] T 5007b09b832a4c70b24cc897caec918d P f0fc0dcbac8f49c19b2e2e12098b287a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:55:57.298395 3129 heartbeater.cc:503] Master 127.20.116.190:43661 was elected leader, sending a full tablet report...
I20260928 14:55:57.298471 3261 heartbeater.cc:503] Master 127.20.116.190:43661 was elected leader, sending a full tablet report...
W20260928 14:55:57.350155 3130 tablet.cc:2369] T 5007b09b832a4c70b24cc897caec918d P f0fc0dcbac8f49c19b2e2e12098b287a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:55:57.376159 3275 raft_consensus.cc:482] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:55:57.376293 3275 raft_consensus.cc:504] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } }
I20260928 14:55:57.376756 3275 leader_election.cc:288] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0fc0dcbac8f49c19b2e2e12098b287a (127.20.116.130:37939), e636a58f85d14261b150d45a4f21bef4 (127.20.116.129:40339)
I20260928 14:55:57.380000 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:57.376922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:37939 (local address 127.20.116.131:44985)
0928 14:55:57.377001 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:57.377013 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:57.377061 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:57.377263 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:57.377264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:57.377266 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:55:57.377354 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:57.377356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.378110 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.378111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:57.379111 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:57.379114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.379730 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.379731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:57.379809 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:57.379813 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:55:57.379873 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 14:55:57.379914 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":54}
I20260928 14:55:57.380003 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:57.377029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:44985 (local address 127.20.116.130:37939)
0928 14:55:57.377118 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:57.377120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:57.377131 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:55:57.377163 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:57.377164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:57.377176 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:57.377256 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:55:57.377444 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.378014 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.379193 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.379648 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.379703 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:57.379710 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:57.379886 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 14:55:57.379922 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 14:55:57.380290 2999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:57.377145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:33083 (local address 127.20.116.129:40339)
0928 14:55:57.377203 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:57.377205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:57.377343 (+ 138us) server_negotiation.cc:407] Connection header received
0928 14:55:57.377376 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:57.377378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:57.377407 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:57.377479 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:55:57.377674 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.378254 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.379382 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.379906 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.379974 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:57.380048 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:57.380164 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 14:55:57.380231 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 14:55:57.380410 3083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5007b09b832a4c70b24cc897caec918d" candidate_uuid: "aef01926632b40e2a3329141e35bd3d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" is_pre_election: true
I20260928 14:55:57.380502 3277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:57.377057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:40339 (local address 127.20.116.131:33083)
0928 14:55:57.377261 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:57.377273 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:57.377337 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:57.377495 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:57.377496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:57.377497 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:55:57.377570 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:57.377572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.378338 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.378339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:57.379308 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:57.379311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.379982 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.379983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:57.380057 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:57.380062 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:55:57.380366 (+ 304us) client_negotiation.cc:240] Negotiation successful
0928 14:55:57.380409 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":118,"threads_started":1}
I20260928 14:55:57.380609 3083 raft_consensus.cc:2454] T 5007b09b832a4c70b24cc897caec918d P f0fc0dcbac8f49c19b2e2e12098b287a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aef01926632b40e2a3329141e35bd3d7 in term 0.
I20260928 14:55:57.380864 3151 leader_election.cc:302] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aef01926632b40e2a3329141e35bd3d7, f0fc0dcbac8f49c19b2e2e12098b287a; no voters:
I20260928 14:55:57.380916 2927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5007b09b832a4c70b24cc897caec918d" candidate_uuid: "aef01926632b40e2a3329141e35bd3d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e636a58f85d14261b150d45a4f21bef4" is_pre_election: true
I20260928 14:55:57.381013 3275 raft_consensus.cc:2781] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:55:57.381070 3275 raft_consensus.cc:482] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:55:57.381067 2927 raft_consensus.cc:2454] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aef01926632b40e2a3329141e35bd3d7 in term 0.
I20260928 14:55:57.381105 3275 raft_consensus.cc:3037] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:55:57.381814 3275 raft_consensus.cc:504] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } }
I20260928 14:55:57.381916 3275 leader_election.cc:288] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7 [CANDIDATE]: Term 1 election: Requested vote from peers f0fc0dcbac8f49c19b2e2e12098b287a (127.20.116.130:37939), e636a58f85d14261b150d45a4f21bef4 (127.20.116.129:40339)
I20260928 14:55:57.382105 3083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5007b09b832a4c70b24cc897caec918d" candidate_uuid: "aef01926632b40e2a3329141e35bd3d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a"
I20260928 14:55:57.382184 3083 raft_consensus.cc:3037] T 5007b09b832a4c70b24cc897caec918d P f0fc0dcbac8f49c19b2e2e12098b287a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:55:57.382150 2927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5007b09b832a4c70b24cc897caec918d" candidate_uuid: "aef01926632b40e2a3329141e35bd3d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e636a58f85d14261b150d45a4f21bef4"
I20260928 14:55:57.382269 2927 raft_consensus.cc:3037] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:55:57.382997 3083 raft_consensus.cc:2454] T 5007b09b832a4c70b24cc897caec918d P f0fc0dcbac8f49c19b2e2e12098b287a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aef01926632b40e2a3329141e35bd3d7 in term 1.
I20260928 14:55:57.383091 2927 raft_consensus.cc:2454] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aef01926632b40e2a3329141e35bd3d7 in term 1.
I20260928 14:55:57.383167 3151 leader_election.cc:302] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aef01926632b40e2a3329141e35bd3d7, f0fc0dcbac8f49c19b2e2e12098b287a; no voters:
I20260928 14:55:57.383275 3275 raft_consensus.cc:2781] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:55:57.383426 3275 raft_consensus.cc:686] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7 [term 1 LEADER]: Becoming Leader. State: Replica: aef01926632b40e2a3329141e35bd3d7, State: Running, Role: LEADER
I20260928 14:55:57.383549 3275 consensus_queue.cc:237] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } }
I20260928 14:55:57.384572 2803 catalog_manager.cc:5495] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to aef01926632b40e2a3329141e35bd3d7 (127.20.116.131). New cstate: current_term: 1 leader_uuid: "aef01926632b40e2a3329141e35bd3d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } health_report { overall_health: UNKNOWN } } }
I20260928 14:55:57.409083 3269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:57.406798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56794 (local address 127.20.116.131:36955)
0928 14:55:57.406864 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:57.406866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:57.406874 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:55:57.406944 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:57.406945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:57.406964 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:57.407026 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:55:57.407129 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.407590 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.408311 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:57.408500 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:57.408558 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:57.408715 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:55:57.408827 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:55:57.408890 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:57.408922 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 14:55:57.408940 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:55:57.411017 2927 raft_consensus.cc:1261] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [term 1 FOLLOWER]: Refusing update from remote peer aef01926632b40e2a3329141e35bd3d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:55:57.411051 3083 raft_consensus.cc:1261] T 5007b09b832a4c70b24cc897caec918d P f0fc0dcbac8f49c19b2e2e12098b287a [term 1 FOLLOWER]: Refusing update from remote peer aef01926632b40e2a3329141e35bd3d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:55:57.411365 3275 consensus_queue.cc:1035] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:55:57.411520 3278 consensus_queue.cc:1035] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:55:57.414350 3282 mvcc.cc:205] Tried to move back new op lower bound from 7334327735952699392 to 7334327735843844096. Current Snapshot: MvccSnapshot[applied={T|T < 7334327735952699392 or (T in {7334327735952699392})}]
I20260928 14:55:57.415946 2802 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:55:57.416049 2802 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:55:57.416800 2802 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316: Sending DeleteTablet for 3 replicas of tablet 5007b09b832a4c70b24cc897caec918d
I20260928 14:55:57.417130 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 2841
I20260928 14:55:57.417253 3195 tablet_service.cc:1514] Processing DeleteTablet for tablet 5007b09b832a4c70b24cc897caec918d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:55:57 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:56790
I20260928 14:55:57.417271 3063 tablet_service.cc:1514] Processing DeleteTablet for tablet 5007b09b832a4c70b24cc897caec918d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:55:57 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:37776
I20260928 14:55:57.417925 3291 tablet_replica.cc:330] T 5007b09b832a4c70b24cc897caec918d P f0fc0dcbac8f49c19b2e2e12098b287a: stopping tablet replica
I20260928 14:55:57.418036 3291 raft_consensus.cc:2229] T 5007b09b832a4c70b24cc897caec918d P f0fc0dcbac8f49c19b2e2e12098b287a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:55:57.418066 3292 tablet_replica.cc:330] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7: stopping tablet replica
I20260928 14:55:57.418148 3291 raft_consensus.cc:2258] T 5007b09b832a4c70b24cc897caec918d P f0fc0dcbac8f49c19b2e2e12098b287a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:55:57.418386 3292 raft_consensus.cc:2229] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:55:57.418509 3292 raft_consensus.cc:2258] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:55:57.418947 3292 ts_tablet_manager.cc:1905] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:55:57.420624 3292 ts_tablet_manager.cc:1918] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:55:57.420737 3292 log.cc:1196] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/5007b09b832a4c70b24cc897caec918d
I20260928 14:55:57.421012 3292 ts_tablet_manager.cc:1939] T 5007b09b832a4c70b24cc897caec918d P aef01926632b40e2a3329141e35bd3d7: Deleting consensus metadata
I20260928 14:55:57.421401 2790 catalog_manager.cc:4841] TS aef01926632b40e2a3329141e35bd3d7 (127.20.116.131:36955): tablet 5007b09b832a4c70b24cc897caec918d (table test-table [id=1bc4eaf9f6274a49ad1790e0197856a2]) successfully deleted
I20260928 14:55:57.422869 3291 ts_tablet_manager.cc:1905] T 5007b09b832a4c70b24cc897caec918d P f0fc0dcbac8f49c19b2e2e12098b287a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:55:57.424218 3291 ts_tablet_manager.cc:1918] T 5007b09b832a4c70b24cc897caec918d P f0fc0dcbac8f49c19b2e2e12098b287a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:55:57.424314 3291 log.cc:1196] T 5007b09b832a4c70b24cc897caec918d P f0fc0dcbac8f49c19b2e2e12098b287a: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/5007b09b832a4c70b24cc897caec918d
I20260928 14:55:57.424566 3291 ts_tablet_manager.cc:1939] T 5007b09b832a4c70b24cc897caec918d P f0fc0dcbac8f49c19b2e2e12098b287a: Deleting consensus metadata
I20260928 14:55:57.424995 2790 catalog_manager.cc:4841] TS f0fc0dcbac8f49c19b2e2e12098b287a (127.20.116.130:37939): tablet 5007b09b832a4c70b24cc897caec918d (table test-table [id=1bc4eaf9f6274a49ad1790e0197856a2]) successfully deleted
W20260928 14:55:57.425520 2790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.116.129:40339 (error 108)
I20260928 14:55:57.425896 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 2996
I20260928 14:55:57.426455 2998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:57.425870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:40339 (local address 127.0.0.1:59518)
0928 14:55:57.426369 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:57.426415 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.129:40339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476}
W20260928 14:55:57.426658 2790 catalog_manager.cc:4584] TS e636a58f85d14261b150d45a4f21bef4 (127.20.116.129:40339): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5007b09b832a4c70b24cc897caec918d: Network error: Client connection negotiation failed: client connection to 127.20.116.129:40339: connect: Connection refused (error 111)
I20260928 14:55:57.431182 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 3132
I20260928 14:55:57.436632 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 2770
I20260928 14:55:57.442152 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:43661
--webserver_interface=127.20.116.190
--webserver_port=38731
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:43661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:55:57.526060 3294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:57.526280 3294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:57.526304 3294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:57.527485 3294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:55:57.527529 3294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:57.527544 3294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:57.527556 3294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:55:57.527568 3294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:55:57.529036 3294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:43661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:43661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=38731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:57.529254 3294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:57.529446 3294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:57.534489 3302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:57.534471 3299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:57.534474 3300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:57.534853 3294 server_base.cc:1034] running on GCE node
I20260928 14:55:57.535128 3294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:57.535445 3294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:57.536628 3294 hybrid_clock.cc:648] HybridClock initialized: now 1790607357536623 us; error 47 us; skew 500 ppm
Sep 28 14:55:57 dist-test-slave-pxl6 krb5kdc[2221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607357, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:57.539283 3294 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:55:57.539737 3294 webserver.cc:460] Webserver started at http://127.20.116.190:38731/ using document root <none> and password file <none>
I20260928 14:55:57.539988 3294 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:57.540035 3294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:55:57.541406 3294 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 14:55:57.542117 3309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:57.542375 3294 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:55:57.542510 3294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "10ee404f27ab49e1b46ca6daa3e49316"
format_stamp: "Formatted at 2026-09-28 14:55:55 on dist-test-slave-pxl6"
server_key: "e9f446703b8b700ec4cf9f0f9990709a"
server_key_iv: "f22354d35db10724207c71d302392f6f"
server_key_version: "encryptionkey@0"
I20260928 14:55:57.543025 3294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:57.564041 3294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:57.566293 3294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:57.566463 3294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:57.570415 3294 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:43661
I20260928 14:55:57.570452 3361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:43661 every 8 connection(s)
I20260928 14:55:57.570870 3294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:55:57.572436 3362 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:55:57.573046 3362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316: Bootstrap starting.
I20260928 14:55:57.577474 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 3294
I20260928 14:55:57.577823 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:40339
--local_ip_for_outbound_sockets=127.20.116.129
--tserver_master_addrs=127.20.116.190:43661
--webserver_port=37947
--webserver_interface=127.20.116.129
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:55:57.582752 3362 log.cc:824] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316: Log is configured to *not* fsync() on all Append() calls
I20260928 14:55:57.585122 3362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:55:57.585455 3362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316: Bootstrap complete.
I20260928 14:55:57.587944 3362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ee404f27ab49e1b46ca6daa3e49316" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43661 } }
I20260928 14:55:57.588165 3362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee404f27ab49e1b46ca6daa3e49316, State: Initialized, Role: FOLLOWER
I20260928 14:55:57.588335 3362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ee404f27ab49e1b46ca6daa3e49316" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43661 } }
I20260928 14:55:57.588395 3362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:55:57.588456 3362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:55:57.588508 3362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 2 FOLLOWER]: Advancing to term 3
I20260928 14:55:57.589517 3362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ee404f27ab49e1b46ca6daa3e49316" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43661 } }
I20260928 14:55:57.589639 3362 leader_election.cc:302] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10ee404f27ab49e1b46ca6daa3e49316; no voters:
I20260928 14:55:57.589866 3362 leader_election.cc:288] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 14:55:57.589905 3366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 14:55:57.590068 3366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [term 3 LEADER]: Becoming Leader. State: Replica: 10ee404f27ab49e1b46ca6daa3e49316, State: Running, Role: LEADER
I20260928 14:55:57.590150 3362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:55:57.590236 3366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ee404f27ab49e1b46ca6daa3e49316" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43661 } }
I20260928 14:55:57.590521 3367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "10ee404f27ab49e1b46ca6daa3e49316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ee404f27ab49e1b46ca6daa3e49316" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43661 } } }
I20260928 14:55:57.590749 3367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [sys.catalog]: This master's current role is: LEADER
I20260928 14:55:57.590521 3366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10ee404f27ab49e1b46ca6daa3e49316. Latest consensus state: current_term: 3 leader_uuid: "10ee404f27ab49e1b46ca6daa3e49316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ee404f27ab49e1b46ca6daa3e49316" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43661 } } }
I20260928 14:55:57.591135 3375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:55:57.591275 3366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10ee404f27ab49e1b46ca6daa3e49316 [sys.catalog]: This master's current role is: LEADER
I20260928 14:55:57.592109 3375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:55:57.592201 3375 catalog_manager.cc:1259] Loaded cluster ID: 7bd2333a531945a78182e96e40f1025c
I20260928 14:55:57.592223 3375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 14:55:57.592445 3375 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6d2e2d59ad google::LogMessage::Fail() at ??:0
@ 0x7f6d2e2d888a google::LogMessage::SendToLog() at ??:0
@ 0x7f6d2e2d54de google::LogMessage::Flush() at ??:0
@ 0x7f6d2e2d6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6d361f60f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f6d361ff8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6d301a30d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6d30198954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6d30ffc6db start_thread at ??:0
@ 0x7f6d2cdd361f clone at ??:0
W20260928 14:55:57.679500 3364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:57.679708 3364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:57.679733 3364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:57.680961 3364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:57.681007 3364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:57.681035 3364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:55:57.682606 3364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:40339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=37947
--tserver_master_addrs=127.20.116.190:43661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:57.682884 3364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:57.683161 3364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:57.688417 3401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:57.688449 3398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:57.688436 3399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:57.688807 3364 server_base.cc:1034] running on GCE node
I20260928 14:55:57.689016 3364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:57.689348 3364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:57.690656 3364 hybrid_clock.cc:648] HybridClock initialized: now 1790607357690632 us; error 29 us; skew 500 ppm
Sep 28 14:55:57 dist-test-slave-pxl6 krb5kdc[2221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607357, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:57.693843 3364 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:55:57.694357 3364 webserver.cc:460] Webserver started at http://127.20.116.129:37947/ using document root <none> and password file <none>
I20260928 14:55:57.694640 3364 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:57.694689 3364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:55:57.696229 3364 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:55:57.696957 3408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:57.697190 3364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:55:57.697276 3364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "e636a58f85d14261b150d45a4f21bef4"
format_stamp: "Formatted at 2026-09-28 14:55:55 on dist-test-slave-pxl6"
server_key: "1daf754ae395c00bd4f6a9f89c2e1c38"
server_key_iv: "2f01511f6b810bbefc038da52d574898"
server_key_version: "encryptionkey@0"
I20260928 14:55:57.697738 3364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:57.718197 3364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:57.721187 3364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:57.721372 3364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:57.721668 3364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:55:57.722226 3415 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 14:55:57.723883 3364 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:55:57.723956 3364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 14:55:57.723994 3364 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:55:57.724960 3364 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:55:57.725011 3364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 14:55:57.725132 3415 tablet_bootstrap.cc:492] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4: Bootstrap starting.
I20260928 14:55:57.732587 3364 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:40339
I20260928 14:55:57.733125 3364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:55:57.734831 3522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:40339 every 8 connection(s)
I20260928 14:55:57.735641 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 3364
I20260928 14:55:57.735960 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:37939
--local_ip_for_outbound_sockets=127.20.116.130
--tserver_master_addrs=127.20.116.190:43661
--webserver_port=45231
--webserver_interface=127.20.116.130
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:55:57.737020 3415 log.cc:824] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:55:57.739573 3415 tablet_bootstrap.cc:492] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:55:57.739893 3415 tablet_bootstrap.cc:492] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4: Bootstrap complete.
I20260928 14:55:57.741180 3415 ts_tablet_manager.cc:1397] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20260928 14:55:57.743276 3415 raft_consensus.cc:348] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } }
I20260928 14:55:57.743731 3415 raft_consensus.cc:729] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e636a58f85d14261b150d45a4f21bef4, State: Initialized, Role: FOLLOWER
I20260928 14:55:57.743942 3415 consensus_queue.cc:260] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } }
I20260928 14:55:57.744331 3415 ts_tablet_manager.cc:1428] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
W20260928 14:55:57.823400 3527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:57.823599 3527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:57.823630 3527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:57.824790 3527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:57.824831 3527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:57.824858 3527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:55:57.826347 3527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:37939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=45231
--tserver_master_addrs=127.20.116.190:43661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:57.826622 3527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:57.826884 3527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:57.832237 3533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:57.832350 3527 server_base.cc:1034] running on GCE node
W20260928 14:55:57.832201 3534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:57.832505 3536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:57.832801 3527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:57.833110 3527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:57.834301 3527 hybrid_clock.cc:648] HybridClock initialized: now 1790607357834287 us; error 44 us; skew 500 ppm
Sep 28 14:55:57 dist-test-slave-pxl6 krb5kdc[2221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607357, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:57.837755 3527 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:55:57.838325 3527 webserver.cc:460] Webserver started at http://127.20.116.130:45231/ using document root <none> and password file <none>
I20260928 14:55:57.838649 3527 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:57.838721 3527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:55:57.840606 3527 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:55:57.841325 3543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:57.841533 3527 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:55:57.841640 3527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "f0fc0dcbac8f49c19b2e2e12098b287a"
format_stamp: "Formatted at 2026-09-28 14:55:56 on dist-test-slave-pxl6"
server_key: "903e1aa1fe5308626bb27cf16cff9382"
server_key_iv: "96379b06d3136d481fe56e9c35c44c0f"
server_key_version: "encryptionkey@0"
I20260928 14:55:57.842098 3527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260928 14:55:57.852768 3525 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:55:57.733928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.129:43983)
0928 14:55:57.734229 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:57.734299 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:57.735317 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:57.852515 (+117198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":109,"threads_started":1}
W20260928 14:55:57.853077 3523 heartbeater.cc:644] Failed to heartbeat to 127.20.116.190:43661 (0 consecutive failures): Network error: Failed to ping master at 127.20.116.190:43661: Client connection negotiation failed: client connection to 127.20.116.190:43661: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:55:57.853467 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:57.853309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.129:34859)
0928 14:55:57.853392 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:57.853441 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 14:55:57.853876 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:57.853766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.129:59829)
0928 14:55:57.853818 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:57.853846 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 14:55:57.853994 3523 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:55:57.854943 3527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:57.857338 3527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:57.857508 3527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:57.857761 3527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:55:57.858079 3527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:55:57.858135 3527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:57.858180 3527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:55:57.858219 3527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:57.864491 3527 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:37939
I20260928 14:55:57.864574 3656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:37939 every 8 connection(s)
I20260928 14:55:57.864952 3527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:55:57.866091 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:57.865688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.130:50239)
0928 14:55:57.865850 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:57.865942 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
W20260928 14:55:57.866500 3657 heartbeater.cc:644] Failed to heartbeat to 127.20.116.190:43661 (0 consecutive failures): Network error: Failed to ping master at 127.20.116.190:43661: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
I20260928 14:55:57.866859 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:57.866754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.130:40725)
0928 14:55:57.866805 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:57.866838 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 14:55:57.867146 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:57.867050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.130:39847)
0928 14:55:57.867092 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:57.867114 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 14:55:57.867245 3657 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:55:57.873528 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 3527
I20260928 14:55:57.873769 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:36955
--local_ip_for_outbound_sockets=127.20.116.131
--tserver_master_addrs=127.20.116.190:43661
--webserver_port=34959
--webserver_interface=127.20.116.131
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:55:57.957581 3660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:55:57.957798 3660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:55:57.957831 3660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:55:57.958961 3660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:55:57.958998 3660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:55:57.959024 3660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:55:57.960438 3660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:36955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=34959
--tserver_master_addrs=127.20.116.190:43661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:55:57.960675 3660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:55:57.960914 3660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:55:57.966005 3668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:57.966022 3666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:55:57.966005 3665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:55:57.966367 3660 server_base.cc:1034] running on GCE node
I20260928 14:55:57.966696 3660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:55:57.966995 3660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:55:57.968191 3660 hybrid_clock.cc:648] HybridClock initialized: now 1790607357968173 us; error 29 us; skew 500 ppm
Sep 28 14:55:57 dist-test-slave-pxl6 krb5kdc[2221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607357, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:55:57.971048 3660 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:55:57.971613 3660 webserver.cc:460] Webserver started at http://127.20.116.131:34959/ using document root <none> and password file <none>
I20260928 14:55:57.971961 3660 fs_manager.cc:359] Metadata directory not provided
I20260928 14:55:57.972038 3660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:55:57.973820 3660 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:55:57.974370 3675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:57.974550 3660 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:55:57.974647 3660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "aef01926632b40e2a3329141e35bd3d7"
format_stamp: "Formatted at 2026-09-28 14:55:56 on dist-test-slave-pxl6"
server_key: "868414ef3772392aa08e33cb95d350fd"
server_key_iv: "f9208c9698ff82ad140eddd593b1590b"
server_key_version: "encryptionkey@0"
I20260928 14:55:57.974978 3660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:55:58.000880 3660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:55:58.003607 3660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:55:58.003747 3660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:55:58.003965 3660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:55:58.004309 3660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:55:58.004346 3660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:58.004370 3660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:55:58.004416 3660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:55:58.010438 3660 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:36955
I20260928 14:55:58.010552 3788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:36955 every 8 connection(s)
I20260928 14:55:58.010960 3660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:55:58.012123 3791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:58.011698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.131:54859)
0928 14:55:58.011862 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:58.011958 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
W20260928 14:55:58.012547 3789 heartbeater.cc:644] Failed to heartbeat to 127.20.116.190:43661 (0 consecutive failures): Network error: Failed to ping master at 127.20.116.190:43661: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
I20260928 14:55:58.012874 3791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:58.012776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.131:41113)
0928 14:55:58.012826 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:58.012855 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 14:55:58.013139 3791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:58.013071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.131:34609)
0928 14:55:58.013092 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:58.013108 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 14:55:58.013255 3789 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:55:58.019112 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 3660
I20260928 14:55:58.854672 3793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:58.854335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.129:50961)
0928 14:55:58.854483 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:58.854523 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260928 14:55:58.867779 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:58.867554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.130:37157)
0928 14:55:58.867686 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:58.867717 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260928 14:55:59.013866 3795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:59.013598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.131:56671)
0928 14:55:59.013747 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:59.013792 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260928 14:55:59.502776 3796 raft_consensus.cc:482] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:55:59.502916 3796 raft_consensus.cc:504] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } }
I20260928 14:55:59.503242 3796 leader_election.cc:288] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0fc0dcbac8f49c19b2e2e12098b287a (127.20.116.130:37939), aef01926632b40e2a3329141e35bd3d7 (127.20.116.131:36955)
Sep 28 14:55:59 dist-test-slave-pxl6 krb5kdc[2221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607357, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.131@KRBTEST.COM
Sep 28 14:55:59 dist-test-slave-pxl6 krb5kdc[2221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607357, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.130@KRBTEST.COM
I20260928 14:55:59.517014 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:59.503882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:33531 (local address 127.20.116.131:36955)
0928 14:55:59.504022 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:59.504028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:59.504094 (+ 66us) server_negotiation.cc:407] Connection header received
0928 14:55:59.504685 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:59.504689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:59.504754 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:59.504923 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:59.506032 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:59.507504 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:59.509037 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:59.509687 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:59.509819 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:59.512992 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:59.512996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:59.512996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:59.513017 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:59.514993 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:59.515694 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:59.515698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:59.515698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:59.515794 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:59.516116 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:59.516117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:59.516118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:59.516377 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:59.516473 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:59.516775 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 14:55:59.516906 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260928 14:55:59.517014 3799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:59.503730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:36955 (local address 127.20.116.129:33531)
0928 14:55:59.503822 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:59.503831 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:59.503889 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:59.505034 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:59.505037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:59.505565 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:59.505887 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:59.505895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:59.507642 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:59.507642 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:59.508931 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:59.508935 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:59.509850 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:59.509851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:59.510043 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:59.510867 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:59.510880 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:59.512871 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:59.515106 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:59.515114 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:59.515116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:59.515578 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:59.515878 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:59.515879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:59.515879 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:59.516046 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:59.516453 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:59.516461 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:59.516610 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 14:55:59.516740 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 14:55:59.516907 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":112,"thread_start_us":49,"threads_started":1}
I20260928 14:55:59.517236 3798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:59.503580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:38719 (local address 127.20.116.130:37939)
0928 14:55:59.503823 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 14:55:59.503839 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:55:59.504078 (+ 239us) server_negotiation.cc:407] Connection header received
0928 14:55:59.504685 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:55:59.504689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:55:59.504754 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:55:59.504923 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:55:59.506032 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:59.507512 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:59.509037 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:55:59.509722 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:55:59.509819 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:59.513300 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:55:59.513305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:55:59.513305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:55:59.513319 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:55:59.515210 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:59.515896 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:59.515909 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:59.515909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:59.516040 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:55:59.516303 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:55:59.516304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:55:59.516304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:55:59.516565 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:55:59.516747 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 14:55:59.516944 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 14:55:59.517090 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":107,"threads_started":1}
I20260928 14:55:59.517474 3797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:59.503466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:37939 (local address 127.20.116.129:38719)
0928 14:55:59.503788 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:59.503811 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 14:55:59.504054 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:55:59.505034 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:55:59.505037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:55:59.505565 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:55:59.505887 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:59.505895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:59.507627 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:59.507628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:59.508931 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:55:59.508935 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:55:59.509848 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:55:59.509850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:55:59.510042 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:55:59.510867 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:55:59.510880 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:55:59.513191 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:55:59.515303 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:59.515304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:59.515304 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:59.515635 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:59.516148 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:55:59.516149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:55:59.516150 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:55:59.516227 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:55:59.516723 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:55:59.516724 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:55:59.516778 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 14:55:59.516873 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 14:55:59.516997 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":36,"thread_start_us":78,"threads_started":1}
W20260928 14:55:59.517912 3412 leader_election.cc:341] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aef01926632b40e2a3329141e35bd3d7 (127.20.116.131:36955): Not found: Tablet not found: 5007b09b832a4c70b24cc897caec918d
W20260928 14:55:59.518329 3412 leader_election.cc:341] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0fc0dcbac8f49c19b2e2e12098b287a (127.20.116.130:37939): Not found: Tablet not found: 5007b09b832a4c70b24cc897caec918d
I20260928 14:55:59.518383 3412 leader_election.cc:302] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e636a58f85d14261b150d45a4f21bef4; no voters: aef01926632b40e2a3329141e35bd3d7, f0fc0dcbac8f49c19b2e2e12098b287a
I20260928 14:55:59.518571 3796 raft_consensus.cc:2726] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:55:59.855294 3797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:59.855123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.129:41043)
0928 14:55:59.855201 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:59.855251 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 14:55:59.868433 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:55:59.868189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.130:39969)
0928 14:55:59.868330 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 14:55:59.868389 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260928 14:56:00.014665 3802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:00.014382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.131:56319)
0928 14:56:00.014514 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:00.014602 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260928 14:56:00.856140 3803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:00.855877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.129:52099)
0928 14:56:00.856037 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:00.856088 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":72,"threads_started":1}
I20260928 14:56:00.869308 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:00.869032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.130:47871)
0928 14:56:00.869208 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:00.869261 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":110,"threads_started":1}
I20260928 14:56:01.015434 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:01.015205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.131:57015)
0928 14:56:01.015346 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:01.015390 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260928 14:56:01.693104 3806 raft_consensus.cc:482] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:56:01.693202 3806 raft_consensus.cc:504] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } }
I20260928 14:56:01.693337 3806 leader_election.cc:288] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0fc0dcbac8f49c19b2e2e12098b287a (127.20.116.130:37939), aef01926632b40e2a3329141e35bd3d7 (127.20.116.131:36955)
W20260928 14:56:01.693877 3412 leader_election.cc:341] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aef01926632b40e2a3329141e35bd3d7 (127.20.116.131:36955): Not found: Tablet not found: 5007b09b832a4c70b24cc897caec918d
W20260928 14:56:01.693951 3412 leader_election.cc:341] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0fc0dcbac8f49c19b2e2e12098b287a (127.20.116.130:37939): Not found: Tablet not found: 5007b09b832a4c70b24cc897caec918d
I20260928 14:56:01.693969 3412 leader_election.cc:302] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e636a58f85d14261b150d45a4f21bef4; no voters: aef01926632b40e2a3329141e35bd3d7, f0fc0dcbac8f49c19b2e2e12098b287a
I20260928 14:56:01.694080 3806 raft_consensus.cc:2726] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:56:01.856887 3807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:01.856654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.129:51211)
0928 14:56:01.856780 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:01.856819 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260928 14:56:01.870152 3808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:01.869843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.130:48023)
0928 14:56:01.870037 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:01.870081 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":109,"threads_started":1}
I20260928 14:56:02.016204 3809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:02.015970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.131:59911)
0928 14:56:02.016110 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:02.016156 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260928 14:56:02.857810 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:02.857531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.129:53995)
0928 14:56:02.857710 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:02.857760 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":87,"threads_started":1}
I20260928 14:56:02.871058 3811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:02.870805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.130:58901)
0928 14:56:02.870958 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:02.871012 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":62,"threads_started":1}
I20260928 14:56:03.017151 3812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:03.016896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.131:35763)
0928 14:56:03.017054 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:03.017102 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
I20260928 14:56:03.858980 3813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:03.858406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.129:45833)
0928 14:56:03.858862 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:03.858930 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":67,"threads_started":1}
I20260928 14:56:03.871770 3814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:03.871558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.130:39133)
0928 14:56:03.871677 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:03.871719 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260928 14:56:04.018092 3815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:04.017786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.131:53725)
0928 14:56:04.017971 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:04.018023 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":85,"threads_started":1}
I20260928 14:56:04.495532 3816 raft_consensus.cc:482] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:56:04.495627 3816 raft_consensus.cc:504] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } }
I20260928 14:56:04.495800 3816 leader_election.cc:288] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0fc0dcbac8f49c19b2e2e12098b287a (127.20.116.130:37939), aef01926632b40e2a3329141e35bd3d7 (127.20.116.131:36955)
W20260928 14:56:04.496183 3412 leader_election.cc:341] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aef01926632b40e2a3329141e35bd3d7 (127.20.116.131:36955): Not found: Tablet not found: 5007b09b832a4c70b24cc897caec918d
W20260928 14:56:04.496263 3412 leader_election.cc:341] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0fc0dcbac8f49c19b2e2e12098b287a (127.20.116.130:37939): Not found: Tablet not found: 5007b09b832a4c70b24cc897caec918d
I20260928 14:56:04.496279 3412 leader_election.cc:302] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e636a58f85d14261b150d45a4f21bef4; no voters: aef01926632b40e2a3329141e35bd3d7, f0fc0dcbac8f49c19b2e2e12098b287a
I20260928 14:56:04.496344 3816 raft_consensus.cc:2726] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:56:04.859793 3817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:04.859508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.129:54913)
0928 14:56:04.859683 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:04.859736 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":93,"threads_started":1}
I20260928 14:56:04.872485 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:04.872236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.130:39773)
0928 14:56:04.872406 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:04.872440 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":109,"threads_started":1}
I20260928 14:56:05.018965 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:05.018714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.131:51707)
0928 14:56:05.018870 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:05.018915 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":77,"threads_started":1}
I20260928 14:56:05.860546 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:05.860322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.129:58023)
0928 14:56:05.860464 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:05.860500 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260928 14:56:05.873270 3821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:05.872986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.130:55681)
0928 14:56:05.873164 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:05.873220 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":113,"threads_started":1}
I20260928 14:56:06.019695 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:06.019467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.131:45011)
0928 14:56:06.019610 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:06.019646 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260928 14:56:06.861338 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:06.861069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.129:55939)
0928 14:56:06.861235 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:06.861286 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":72,"threads_started":1}
I20260928 14:56:06.874069 3824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:06.873809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.130:40649)
0928 14:56:06.873984 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:06.874020 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":104,"threads_started":1}
I20260928 14:56:07.020524 3825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:07.020248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.131:39917)
0928 14:56:07.020416 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:07.020473 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":83,"threads_started":1}
I20260928 14:56:07.862172 3826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:07.861909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.129:36503)
0928 14:56:07.862072 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:07.862116 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":69,"threads_started":1}
I20260928 14:56:07.874807 3827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:07.874525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.130:42181)
0928 14:56:07.874702 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:07.874761 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260928 14:56:08.021289 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:08.021049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.131:44489)
0928 14:56:08.021195 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:08.021238 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":66,"threads_started":1}
I20260928 14:56:08.192513 3829 raft_consensus.cc:482] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:56:08.192606 3829 raft_consensus.cc:504] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0fc0dcbac8f49c19b2e2e12098b287a" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 37939 } } peers { permanent_uuid: "aef01926632b40e2a3329141e35bd3d7" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36955 } } peers { permanent_uuid: "e636a58f85d14261b150d45a4f21bef4" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 40339 } }
I20260928 14:56:08.192770 3829 leader_election.cc:288] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0fc0dcbac8f49c19b2e2e12098b287a (127.20.116.130:37939), aef01926632b40e2a3329141e35bd3d7 (127.20.116.131:36955)
W20260928 14:56:08.193159 3412 leader_election.cc:341] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0fc0dcbac8f49c19b2e2e12098b287a (127.20.116.130:37939): Not found: Tablet not found: 5007b09b832a4c70b24cc897caec918d
W20260928 14:56:08.193240 3412 leader_election.cc:341] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aef01926632b40e2a3329141e35bd3d7 (127.20.116.131:36955): Not found: Tablet not found: 5007b09b832a4c70b24cc897caec918d
I20260928 14:56:08.193257 3412 leader_election.cc:302] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e636a58f85d14261b150d45a4f21bef4; no voters: aef01926632b40e2a3329141e35bd3d7, f0fc0dcbac8f49c19b2e2e12098b287a
I20260928 14:56:08.193347 3829 raft_consensus.cc:2726] T 5007b09b832a4c70b24cc897caec918d P e636a58f85d14261b150d45a4f21bef4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:56:08.863013 3830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:08.862759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.129:36211)
0928 14:56:08.862915 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:08.862964 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":65,"threads_started":1}
I20260928 14:56:08.875725 3831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:08.875499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.130:46065)
0928 14:56:08.875633 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:08.875665 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":67,"threads_started":1}
I20260928 14:56:09.022032 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:09.021779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.131:33859)
0928 14:56:09.021941 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:09.021984 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":85,"threads_started":1}
I20260928 14:56:09.863801 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:09.863537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.129:54123)
0928 14:56:09.863703 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:09.863753 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":90,"threads_started":1}
I20260928 14:56:09.876598 3834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:09.876330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.130:42147)
0928 14:56:09.876474 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:09.876524 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260928 14:56:10.022903 3835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:10.022655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.131:44789)
0928 14:56:10.022807 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:10.022851 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
I20260928 14:56:10.864800 3836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:10.864417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.129:37033)
0928 14:56:10.864639 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:10.864714 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":102,"threads_started":1}
I20260928 14:56:10.877429 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:10.877168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.130:60007)
0928 14:56:10.877322 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:10.877378 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260928 14:56:11.023738 3838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:11.023490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.131:43335)
0928 14:56:11.023649 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:11.023693 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":110,"threads_started":1}
I20260928 14:56:11.865636 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:11.865384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.129:37373)
0928 14:56:11.865535 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:11.865583 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260928 14:56:11.878358 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:11.878017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.130:59505)
0928 14:56:11.878178 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:11.878294 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260928 14:56:12.024549 3841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:12.024276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.131:55199)
0928 14:56:12.024445 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:12.024496 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
I20260928 14:56:12.866657 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:12.866378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.129:48963)
0928 14:56:12.866574 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:12.866615 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":98,"threads_started":1}
I20260928 14:56:12.879140 3843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:12.878925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.130:37667)
0928 14:56:12.879055 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:12.879093 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260928 14:56:13.022624 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 3364
I20260928 14:56:13.025430 3844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:13.025206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43661 (local address 127.20.116.131:42561)
0928 14:56:13.025340 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:13.025383 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260928 14:56:13.027967 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 3527
I20260928 14:56:13.032647 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 3660
2026-09-28T14:56:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18566 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:56:13 dist-test-slave-pxl6 krb5kdc[3847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:56:13 dist-test-slave-pxl6 krb5kdc[3847](info): set up 2 sockets
Sep 28 14:56:13 dist-test-slave-pxl6 krb5kdc[3847](info): commencing operation
krb5kdc: starting...
W20260928 14:56:14.083801 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:56:14 dist-test-slave-pxl6 krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:56:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:56:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:56:14.208981 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:40571
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:40571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:14.292340 3860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:14.292579 3860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:14.292601 3860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:14.293833 3860 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 14:56:14.293880 3860 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 14:56:14.293893 3860 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 14:56:14.293905 3860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:56:14.293962 3860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:14.293982 3860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:14.294006 3860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:56:14.294026 3860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:56:14.295624 3860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:40571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:40571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:56:14.295892 3860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:14.296132 3860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:56:14.301502 3860 server_base.cc:1034] running on GCE node
W20260928 14:56:14.301473 3868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:14.301522 3866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:14.301473 3865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:14.301975 3860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:14.302232 3860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:14.303375 3860 hybrid_clock.cc:648] HybridClock initialized: now 1790607374303354 us; error 39 us; skew 500 ppm
Sep 28 14:56:14 dist-test-slave-pxl6 krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607374, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:14.306056 3860 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:56:14.306617 3860 webserver.cc:460] Webserver started at http://127.20.116.190:33869/ using document root <none> and password file <none>
I20260928 14:56:14.306885 3860 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:14.306955 3860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:14.307111 3860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:14.308488 3860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "b0727c1267154abcaedf4f5d90513273"
format_stamp: "Formatted at 2026-09-28 14:56:14 on dist-test-slave-pxl6"
server_key: "c437ae2a158cec7faf9ab49f73bd0e4e"
server_key_iv: "d228d2c4bd100cf8b76daa5d0c37242a"
server_key_version: "encryptionkey@0"
I20260928 14:56:14.308925 3860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "b0727c1267154abcaedf4f5d90513273"
format_stamp: "Formatted at 2026-09-28 14:56:14 on dist-test-slave-pxl6"
server_key: "c437ae2a158cec7faf9ab49f73bd0e4e"
server_key_iv: "d228d2c4bd100cf8b76daa5d0c37242a"
server_key_version: "encryptionkey@0"
I20260928 14:56:14.310386 3860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:56:14.311244 3875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:14.311434 3860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:56:14.311530 3860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "b0727c1267154abcaedf4f5d90513273"
format_stamp: "Formatted at 2026-09-28 14:56:14 on dist-test-slave-pxl6"
server_key: "c437ae2a158cec7faf9ab49f73bd0e4e"
server_key_iv: "d228d2c4bd100cf8b76daa5d0c37242a"
server_key_version: "encryptionkey@0"
I20260928 14:56:14.311614 3860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:14.335973 3860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:14.338423 3860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:14.338606 3860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:14.342582 3860 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:40571
I20260928 14:56:14.342669 3927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:40571 every 8 connection(s)
I20260928 14:56:14.343005 3860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:56:14.344010 3928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:14.344313 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 3860
I20260928 14:56:14.344399 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:56:14.344592 20946 external_mini_cluster.cc:1428] Setting key ee1d84003fa6c65585b09eb559972464
I20260928 14:56:14.347285 3928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273: Bootstrap starting.
I20260928 14:56:14.348028 3928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:14.348346 3928 log.cc:824] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:14.349161 3928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273: No bootstrap required, opened a new log
I20260928 14:56:14.351387 3928 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0727c1267154abcaedf4f5d90513273" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40571 } }
I20260928 14:56:14.351574 3928 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:14.351613 3928 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0727c1267154abcaedf4f5d90513273, State: Initialized, Role: FOLLOWER
I20260928 14:56:14.351769 3928 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0727c1267154abcaedf4f5d90513273" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40571 } }
Sep 28 14:56:14 dist-test-slave-pxl6 krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:56:14.351854 3928 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:56:14.351892 3928 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:56:14.351979 3928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:14.352744 3928 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0727c1267154abcaedf4f5d90513273" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40571 } }
I20260928 14:56:14.352876 3928 leader_election.cc:302] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0727c1267154abcaedf4f5d90513273; no voters:
I20260928 14:56:14.353056 3928 leader_election.cc:288] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:56:14.353118 3933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:14.353353 3928 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:56:14.353304 3933 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 1 LEADER]: Becoming Leader. State: Replica: b0727c1267154abcaedf4f5d90513273, State: Running, Role: LEADER
I20260928 14:56:14.353710 3933 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0727c1267154abcaedf4f5d90513273" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40571 } }
I20260928 14:56:14.354146 3934 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0727c1267154abcaedf4f5d90513273" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0727c1267154abcaedf4f5d90513273" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40571 } } }
I20260928 14:56:14.354236 3934 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:14.354445 3942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:56:14.355057 3942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:56:14.355825 3935 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0727c1267154abcaedf4f5d90513273. Latest consensus state: current_term: 1 leader_uuid: "b0727c1267154abcaedf4f5d90513273" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0727c1267154abcaedf4f5d90513273" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40571 } } }
I20260928 14:56:14.355912 3935 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:14.356704 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:14.345169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.20.116.190:40571)
0928 14:56:14.345408 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:14.345419 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:14.345461 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:56:14.346002 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:14.346012 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:14.346081 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:14.346252 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:14.346904 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.348257 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.349037 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.349641 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.349762 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:14.352383 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:14.352388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:14.352388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:14.352401 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:14.354739 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:14.355220 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:14.355224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:14.355225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:14.355340 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:14.355594 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:14.355596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:14.355596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:14.355933 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:14.356124 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:14.356365 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 14:56:14.356522 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":113,"threads_started":1}
I20260928 14:56:14.359694 3942 catalog_manager.cc:1347] Generated new cluster ID: c9d0ebc525e84a20b82a59d8ab138824
I20260928 14:56:14.359774 3942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:56:14.382797 3942 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:56:14.383409 3942 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:56:14.385349 3942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273: Generated new TSK 0
I20260928 14:56:14.385565 3942 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
I20260928 14:56:14.414069 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:40571
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:14.497910 3954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:14.498112 3954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:14.498137 3954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:14.499398 3954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:14.499445 3954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:14.499472 3954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:56:14.500880 3954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:40571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:56:14.501080 3954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:14.501266 3954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:14.506357 3960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:14.506357 3959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:14.506392 3962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:14.507119 3954 server_base.cc:1034] running on GCE node
I20260928 14:56:14.507356 3954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:14.507673 3954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:14.508859 3954 hybrid_clock.cc:648] HybridClock initialized: now 1790607374508838 us; error 42 us; skew 500 ppm
Sep 28 14:56:14 dist-test-slave-pxl6 krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607374, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:14.511554 3954 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:56:14.512004 3954 webserver.cc:460] Webserver started at http://127.20.116.129:34421/ using document root <none> and password file <none>
I20260928 14:56:14.512194 3954 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:14.512236 3954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:14.512336 3954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:14.513361 3954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "42cafa23b6804de7a072f74483c2719c"
format_stamp: "Formatted at 2026-09-28 14:56:14 on dist-test-slave-pxl6"
server_key: "c82fd90ba3a4cd311c37d65de7f438b9"
server_key_iv: "6f1fe9bcc9e14ccada892d2a6182d4e7"
server_key_version: "encryptionkey@0"
I20260928 14:56:14.513676 3954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "42cafa23b6804de7a072f74483c2719c"
format_stamp: "Formatted at 2026-09-28 14:56:14 on dist-test-slave-pxl6"
server_key: "c82fd90ba3a4cd311c37d65de7f438b9"
server_key_iv: "6f1fe9bcc9e14ccada892d2a6182d4e7"
server_key_version: "encryptionkey@0"
I20260928 14:56:14.515164 3954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:56:14.516018 3969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:14.516245 3954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:56:14.516330 3954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "42cafa23b6804de7a072f74483c2719c"
format_stamp: "Formatted at 2026-09-28 14:56:14 on dist-test-slave-pxl6"
server_key: "c82fd90ba3a4cd311c37d65de7f438b9"
server_key_iv: "6f1fe9bcc9e14ccada892d2a6182d4e7"
server_key_version: "encryptionkey@0"
I20260928 14:56:14.516388 3954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:14.542469 3954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:14.544855 3954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:14.544981 3954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:14.545194 3954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:14.545552 3954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:14.545615 3954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:14.545650 3954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:14.545666 3954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:14.551482 3954 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:35713
I20260928 14:56:14.551594 4083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:35713 every 8 connection(s)
I20260928 14:56:14.551901 3954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:56:14 dist-test-slave-pxl6 krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607374, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:56:14.559929 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 3954
I20260928 14:56:14.560343 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:56:14.560604 20946 external_mini_cluster.cc:1428] Setting key e205f321898ee71b361dfc77cdde1293
I20260928 14:56:14.562578 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:14.552706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:33835 (local address 127.20.116.190:40571)
0928 14:56:14.552792 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:14.552795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:14.552993 (+ 198us) server_negotiation.cc:407] Connection header received
0928 14:56:14.553559 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:14.553561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:14.553571 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:14.553642 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:14.554830 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.555421 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.556517 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.556773 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.556842 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:14.559170 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:14.559173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:14.559173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:14.559182 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:14.560940 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:14.561464 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:14.561466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:14.561466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:14.561540 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:14.561846 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:14.561847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:14.561848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:14.561979 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:14.562058 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:14.562362 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 14:56:14.562460 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:56:14.562906 4086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:14.552657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.129:33835)
0928 14:56:14.552860 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:14.552899 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:14.553479 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:14.553701 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:14.553704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:14.554310 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:14.554689 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:14.554696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.555520 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.555521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:14.556414 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:14.556417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.556881 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.556882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:14.557001 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:14.557530 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:14.557539 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:14.559068 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:14.561055 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:14.561063 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:14.561065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:14.561367 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:14.561617 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:14.561618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:14.561619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:14.561765 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:14.562049 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:14.562057 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:14.562213 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 14:56:14.562325 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 14:56:14.562500 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":119,"threads_started":1}
I20260928 14:56:14.563416 4084 heartbeater.cc:347] Connected to a master server at 127.20.116.190:40571
I20260928 14:56:14.563536 4084 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:14.563732 4084 heartbeater.cc:511] Master 127.20.116.190:40571 requested a full tablet report, sending...
I20260928 14:56:14.564306 3892 ts_manager.cc:194] Registered new tserver with Master: 42cafa23b6804de7a072f74483c2719c (127.20.116.129:35713)
I20260928 14:56:14.565256 3892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:33835
WARNING: no policy specified for kudu/127.20.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
I20260928 14:56:14.589074 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:40571
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:14.671526 4089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:14.671758 4089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:14.671784 4089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:14.672986 4089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:14.673030 4089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:14.673058 4089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:56:14.674657 4089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:40571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:56:14.674940 4089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:14.675191 4089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:14.680493 4094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:14.680503 4095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:14.680557 4097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:14.680823 4089 server_base.cc:1034] running on GCE node
I20260928 14:56:14.681037 4089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:14.681284 4089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:14.682458 4089 hybrid_clock.cc:648] HybridClock initialized: now 1790607374682433 us; error 42 us; skew 500 ppm
Sep 28 14:56:14 dist-test-slave-pxl6 krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607374, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:14.685061 4089 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:56:14.685480 4089 webserver.cc:460] Webserver started at http://127.20.116.130:33645/ using document root <none> and password file <none>
I20260928 14:56:14.685688 4089 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:14.685731 4089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:14.685822 4089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:14.686939 4089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "8f36604068874bc6a3eb9c379212b538"
format_stamp: "Formatted at 2026-09-28 14:56:14 on dist-test-slave-pxl6"
server_key: "cf5752ba887b8ec6c13f62a2763b3096"
server_key_iv: "e5017b0931bf38045b0ebcf9c25f833a"
server_key_version: "encryptionkey@0"
I20260928 14:56:14.687263 4089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "8f36604068874bc6a3eb9c379212b538"
format_stamp: "Formatted at 2026-09-28 14:56:14 on dist-test-slave-pxl6"
server_key: "cf5752ba887b8ec6c13f62a2763b3096"
server_key_iv: "e5017b0931bf38045b0ebcf9c25f833a"
server_key_version: "encryptionkey@0"
I20260928 14:56:14.688653 4089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:56:14.689391 4104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:14.689733 4089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:56:14.689836 4089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "8f36604068874bc6a3eb9c379212b538"
format_stamp: "Formatted at 2026-09-28 14:56:14 on dist-test-slave-pxl6"
server_key: "cf5752ba887b8ec6c13f62a2763b3096"
server_key_iv: "e5017b0931bf38045b0ebcf9c25f833a"
server_key_version: "encryptionkey@0"
I20260928 14:56:14.689916 4089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:14.708014 4089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:14.709936 4089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:14.710095 4089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:14.710330 4089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:14.710757 4089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:14.710811 4089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:14.710855 4089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:14.710881 4089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:14.717760 4089 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:44095
I20260928 14:56:14.717823 4219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:44095 every 8 connection(s)
I20260928 14:56:14.718159 4089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:56:14 dist-test-slave-pxl6 krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607374, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:56:14.724542 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 4089
I20260928 14:56:14.724644 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:56:14.724911 20946 external_mini_cluster.cc:1428] Setting key e57d7890a251a4eceb1548885c111abc
I20260928 14:56:14.728098 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:14.718972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:57701 (local address 127.20.116.190:40571)
0928 14:56:14.719048 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:14.719049 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:14.719269 (+ 220us) server_negotiation.cc:407] Connection header received
0928 14:56:14.719796 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:14.719798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:14.719809 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:14.719877 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:14.720892 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.721394 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.722375 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.722631 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.722699 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:14.724850 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:14.724851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:14.724852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:14.724860 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:14.726678 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:14.727074 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:14.727075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:14.727076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:14.727145 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:14.727485 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:14.727486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:14.727486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:14.727590 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:14.727643 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:14.727973 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 14:56:14.728035 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:56:14.728432 4222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:14.718949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.130:57701)
0928 14:56:14.719135 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:14.719183 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:14.719681 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:14.719940 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:14.719943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:14.720570 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:14.720798 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:14.720805 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.721477 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.721478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:14.722293 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:14.722296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.722715 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.722716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:14.722851 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:14.723303 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:14.723310 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:14.724728 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:14.726781 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:14.726788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:14.726790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:14.727001 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:14.727250 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:14.727251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:14.727251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:14.727412 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:14.727655 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:14.727662 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:14.727832 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 14:56:14.727955 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 14:56:14.728133 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":134,"threads_started":1}
I20260928 14:56:14.728964 4220 heartbeater.cc:347] Connected to a master server at 127.20.116.190:40571
I20260928 14:56:14.729079 4220 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:14.729282 4220 heartbeater.cc:511] Master 127.20.116.190:40571 requested a full tablet report, sending...
I20260928 14:56:14.729724 3892 ts_manager.cc:194] Registered new tserver with Master: 8f36604068874bc6a3eb9c379212b538 (127.20.116.130:44095)
I20260928 14:56:14.730108 3892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:57701
WARNING: no policy specified for kudu/127.20.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
I20260928 14:56:14.753996 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:40571
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:14.836266 4225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:14.836486 4225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:14.836526 4225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:14.837671 4225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:14.837736 4225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:14.837781 4225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:56:14.839332 4225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:40571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:56:14.839581 4225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:14.839838 4225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:14.844731 4233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:14.844733 4230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:14.844728 4231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:14.845134 4225 server_base.cc:1034] running on GCE node
I20260928 14:56:14.845350 4225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:14.845611 4225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:14.846769 4225 hybrid_clock.cc:648] HybridClock initialized: now 1790607374846736 us; error 44 us; skew 500 ppm
Sep 28 14:56:14 dist-test-slave-pxl6 krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607374, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:14.849121 4225 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:56:14.849531 4225 webserver.cc:460] Webserver started at http://127.20.116.131:44487/ using document root <none> and password file <none>
I20260928 14:56:14.849758 4225 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:14.849825 4225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:14.849973 4225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:56:14.851059 4225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "e7183525a8e942e390cc8b570a31e2fe"
format_stamp: "Formatted at 2026-09-28 14:56:14 on dist-test-slave-pxl6"
server_key: "c0b891636334f7684bbf6e9d4dbfadc0"
server_key_iv: "2fe9eda5f9ec2088f36eba87c361e6bb"
server_key_version: "encryptionkey@0"
I20260928 14:56:14.851410 4225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "e7183525a8e942e390cc8b570a31e2fe"
format_stamp: "Formatted at 2026-09-28 14:56:14 on dist-test-slave-pxl6"
server_key: "c0b891636334f7684bbf6e9d4dbfadc0"
server_key_iv: "2fe9eda5f9ec2088f36eba87c361e6bb"
server_key_version: "encryptionkey@0"
I20260928 14:56:14.852722 4225 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:56:14.853502 4240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:14.853698 4225 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:56:14.853822 4225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "e7183525a8e942e390cc8b570a31e2fe"
format_stamp: "Formatted at 2026-09-28 14:56:14 on dist-test-slave-pxl6"
server_key: "c0b891636334f7684bbf6e9d4dbfadc0"
server_key_iv: "2fe9eda5f9ec2088f36eba87c361e6bb"
server_key_version: "encryptionkey@0"
I20260928 14:56:14.853894 4225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:14.889516 4225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:14.891788 4225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:14.891954 4225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:14.892206 4225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:14.892524 4225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:14.892578 4225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:14.892622 4225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:14.892656 4225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:14.898631 4225 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:37981
I20260928 14:56:14.898674 4354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:37981 every 8 connection(s)
I20260928 14:56:14.899003 4225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:56:14.899296 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 4225
I20260928 14:56:14.899410 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:56:14.899610 20946 external_mini_cluster.cc:1428] Setting key ea92bb49491edd42619544b7679587ea
Sep 28 14:56:14 dist-test-slave-pxl6 krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607374, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:56:14.908030 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:14.899808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:35061 (local address 127.20.116.190:40571)
0928 14:56:14.899863 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:14.899865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:14.900089 (+ 224us) server_negotiation.cc:407] Connection header received
0928 14:56:14.900774 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:14.900775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:14.900784 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:14.900827 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:14.901750 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.902235 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.903211 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.903416 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.903473 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:14.905578 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:14.905580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:14.905580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:14.905587 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:14.906884 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:14.907232 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:14.907233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:14.907233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:14.907290 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:14.907534 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:14.907535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:14.907535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:14.907631 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:14.907680 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:14.907907 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 14:56:14.907967 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:56:14.908267 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:14.899791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.131:35061)
0928 14:56:14.899961 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:14.900010 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:14.900689 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:14.900889 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:14.900892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:14.901481 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:14.901677 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:14.901682 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.902319 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.902319 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:14.903133 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:14.903135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.903491 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.903491 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:14.903591 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:14.904126 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:14.904134 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:14.905491 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:14.906950 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:14.906956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:14.906958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:14.907169 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:14.907364 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:14.907365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:14.907365 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:14.907479 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:14.907686 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:14.907691 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:14.907800 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 14:56:14.907880 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 14:56:14.907992 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
I20260928 14:56:14.908800 4355 heartbeater.cc:347] Connected to a master server at 127.20.116.190:40571
I20260928 14:56:14.908905 4355 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:14.909096 4355 heartbeater.cc:511] Master 127.20.116.190:40571 requested a full tablet report, sending...
I20260928 14:56:14.909492 3892 ts_manager.cc:194] Registered new tserver with Master: e7183525a8e942e390cc8b570a31e2fe (127.20.116.131:37981)
I20260928 14:56:14.909888 3892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:35061
I20260928 14:56:14.911360 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:56:14.918565 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:14.912640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38160 (local address 127.20.116.190:40571)
0928 14:56:14.912677 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:14.912678 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:14.912828 (+ 150us) server_negotiation.cc:407] Connection header received
0928 14:56:14.912896 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:14.912897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:14.912907 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:14.912982 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:14.913521 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.913998 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.914815 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.915004 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.915092 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:14.916182 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:14.916183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:14.916183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:14.916191 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:14.917398 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:14.917732 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:14.917733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:14.917733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:14.917791 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:14.917992 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:14.917993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:14.917994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:14.918136 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:14.918252 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:14.918356 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 14:56:14.918430 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 14:56:14.919955 3892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:56:14.921553 3892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:56:14.927608 4368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:14.923903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:35713 (local address 127.0.0.1:44752)
0928 14:56:14.924256 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:14.924303 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:14.924373 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:14.924615 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:14.924618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:14.924621 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:14.924700 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:14.924702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.925645 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.925645 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:14.926425 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:14.926427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.927287 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.927288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:14.927370 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:14.927384 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 14:56:14.927481 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 14:56:14.927538 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":59,"threads_started":1}
I20260928 14:56:14.927639 4370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:14.924171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44752 (local address 127.20.116.129:35713)
0928 14:56:14.924339 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:14.924343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:14.924354 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:56:14.924493 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:14.924495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:14.924511 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:14.924581 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:14.924798 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.925550 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.926515 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.927198 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.927290 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:14.927349 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:14.927510 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 14:56:14.927558 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260928 14:56:14.928493 4369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:14.924059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37328 (local address 127.20.116.131:37981)
0928 14:56:14.924229 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:14.924234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:14.924440 (+ 206us) server_negotiation.cc:407] Connection header received
0928 14:56:14.924512 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:14.924515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:14.924530 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:14.924632 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:14.924905 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.926010 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.927441 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.928103 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.928205 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:14.928257 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:14.928375 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 14:56:14.928418 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
I20260928 14:56:14.928387 4017 tablet_service.cc:1467] Processing CreateTablet for tablet d790174725d54f93a875de8eb60bf53f (DEFAULT_TABLE table=test-table [id=dd84fe1d77c7439baceb42c707c9f894]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:14.928642 4367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:14.923931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:44095 (local address 127.0.0.1:35050)
0928 14:56:14.924704 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:14.924714 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:14.924760 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:14.925039 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:14.925040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:14.925042 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:14.925082 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:14.925084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.926315 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.926316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:14.927412 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:14.927414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.928361 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.928362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:14.928427 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:14.928431 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:56:14.928483 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 14:56:14.928521 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":74,"threads_started":1}
I20260928 14:56:14.928771 4366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:14.923912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:37981 (local address 127.0.0.1:37328)
0928 14:56:14.924347 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:14.924358 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:14.924443 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:14.924760 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:14.924761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:14.924763 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:14.924814 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:14.924815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.926122 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.926123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:14.927322 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:14.927325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.928205 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.928206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:14.928269 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:14.928273 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:56:14.928591 (+ 318us) client_negotiation.cc:240] Negotiation successful
0928 14:56:14.928637 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":10,"thread_start_us":87,"threads_started":1}
I20260928 14:56:14.928776 4017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d790174725d54f93a875de8eb60bf53f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:14.929198 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:14.924147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35050 (local address 127.20.116.130:44095)
0928 14:56:14.924444 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:14.924449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:14.924820 (+ 371us) server_negotiation.cc:407] Connection header received
0928 14:56:14.924869 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:14.924872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:14.924889 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:14.924957 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:14.925160 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.926203 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.927497 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.928189 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.928271 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:14.928324 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:14.928816 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 14:56:14.929092 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":111,"threads_started":1}
I20260928 14:56:14.929451 4288 tablet_service.cc:1467] Processing CreateTablet for tablet d790174725d54f93a875de8eb60bf53f (DEFAULT_TABLE table=test-table [id=dd84fe1d77c7439baceb42c707c9f894]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:14.929780 4288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d790174725d54f93a875de8eb60bf53f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:14.931704 4372 tablet_bootstrap.cc:492] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c: Bootstrap starting.
I20260928 14:56:14.932123 4153 tablet_service.cc:1467] Processing CreateTablet for tablet d790174725d54f93a875de8eb60bf53f (DEFAULT_TABLE table=test-table [id=dd84fe1d77c7439baceb42c707c9f894]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:14.932384 4372 tablet_bootstrap.cc:654] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:14.932441 4153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d790174725d54f93a875de8eb60bf53f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:14.932420 4373 tablet_bootstrap.cc:492] T d790174725d54f93a875de8eb60bf53f P e7183525a8e942e390cc8b570a31e2fe: Bootstrap starting.
I20260928 14:56:14.932992 4372 log.cc:824] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:14.933429 4373 tablet_bootstrap.cc:654] T d790174725d54f93a875de8eb60bf53f P e7183525a8e942e390cc8b570a31e2fe: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:14.933835 4372 tablet_bootstrap.cc:492] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c: No bootstrap required, opened a new log
I20260928 14:56:14.933915 4372 ts_tablet_manager.cc:1397] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:56:14.934046 4373 log.cc:824] T d790174725d54f93a875de8eb60bf53f P e7183525a8e942e390cc8b570a31e2fe: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:14.934791 4373 tablet_bootstrap.cc:492] T d790174725d54f93a875de8eb60bf53f P e7183525a8e942e390cc8b570a31e2fe: No bootstrap required, opened a new log
I20260928 14:56:14.934873 4373 ts_tablet_manager.cc:1397] T d790174725d54f93a875de8eb60bf53f P e7183525a8e942e390cc8b570a31e2fe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:56:14.935983 4376 tablet_bootstrap.cc:492] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538: Bootstrap starting.
I20260928 14:56:14.936170 4372 raft_consensus.cc:348] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:14.936324 4372 raft_consensus.cc:374] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:14.936352 4372 raft_consensus.cc:729] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cafa23b6804de7a072f74483c2719c, State: Initialized, Role: FOLLOWER
I20260928 14:56:14.936468 4372 consensus_queue.cc:260] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:14.936770 4084 heartbeater.cc:503] Master 127.20.116.190:40571 was elected leader, sending a full tablet report...
I20260928 14:56:14.936956 4376 tablet_bootstrap.cc:654] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:14.936761 4372 ts_tablet_manager.cc:1428] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:56:14.936913 4373 raft_consensus.cc:348] T d790174725d54f93a875de8eb60bf53f P e7183525a8e942e390cc8b570a31e2fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:14.937091 4373 raft_consensus.cc:374] T d790174725d54f93a875de8eb60bf53f P e7183525a8e942e390cc8b570a31e2fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:14.937141 4373 raft_consensus.cc:729] T d790174725d54f93a875de8eb60bf53f P e7183525a8e942e390cc8b570a31e2fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7183525a8e942e390cc8b570a31e2fe, State: Initialized, Role: FOLLOWER
I20260928 14:56:14.937294 4373 consensus_queue.cc:260] T d790174725d54f93a875de8eb60bf53f P e7183525a8e942e390cc8b570a31e2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:14.937615 4373 ts_tablet_manager.cc:1428] T d790174725d54f93a875de8eb60bf53f P e7183525a8e942e390cc8b570a31e2fe: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 14:56:14.937702 4376 log.cc:824] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:14.937734 4355 heartbeater.cc:503] Master 127.20.116.190:40571 was elected leader, sending a full tablet report...
I20260928 14:56:14.938549 4376 tablet_bootstrap.cc:492] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538: No bootstrap required, opened a new log
I20260928 14:56:14.938620 4376 ts_tablet_manager.cc:1397] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:56:14.940124 4376 raft_consensus.cc:348] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:14.940258 4376 raft_consensus.cc:374] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:14.940281 4376 raft_consensus.cc:729] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f36604068874bc6a3eb9c379212b538, State: Initialized, Role: FOLLOWER
I20260928 14:56:14.940407 4376 consensus_queue.cc:260] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:14.940696 4376 ts_tablet_manager.cc:1428] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:56:14.940729 4220 heartbeater.cc:503] Master 127.20.116.190:40571 was elected leader, sending a full tablet report...
W20260928 14:56:14.968804 4221 tablet.cc:2369] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:56:14.969812 4377 raft_consensus.cc:482] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:56:14.969920 4377 raft_consensus.cc:504] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:14.970237 4377 leader_election.cc:288] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7183525a8e942e390cc8b570a31e2fe (127.20.116.131:37981), 8f36604068874bc6a3eb9c379212b538 (127.20.116.130:44095)
I20260928 14:56:14.974491 4381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:14.970444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:37981 (local address 127.20.116.129:56851)
0928 14:56:14.970714 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:14.970722 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:14.970798 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:14.970917 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:14.970917 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:14.970919 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:14.970999 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:14.971001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.971889 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.971889 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:14.973341 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:14.973344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.974174 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.974175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:14.974248 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:14.974254 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:56:14.974312 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 14:56:14.974377 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":55,"thread_start_us":141,"threads_started":1}
I20260928 14:56:14.974496 4369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:14.970570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:56851 (local address 127.20.116.131:37981)
0928 14:56:14.970627 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:14.970629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:14.970788 (+ 159us) server_negotiation.cc:407] Connection header received
0928 14:56:14.970854 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:14.970855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:14.970865 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:14.970916 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:14.971074 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.971786 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.973446 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.974066 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.974151 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:14.974172 (+ 21us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:14.974343 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 14:56:14.974399 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 14:56:14.974891 4308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d790174725d54f93a875de8eb60bf53f" candidate_uuid: "42cafa23b6804de7a072f74483c2719c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7183525a8e942e390cc8b570a31e2fe" is_pre_election: true
I20260928 14:56:14.975065 4308 raft_consensus.cc:2454] T d790174725d54f93a875de8eb60bf53f P e7183525a8e942e390cc8b570a31e2fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cafa23b6804de7a072f74483c2719c in term 0.
I20260928 14:56:14.975355 3973 leader_election.cc:302] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cafa23b6804de7a072f74483c2719c, e7183525a8e942e390cc8b570a31e2fe; no voters:
I20260928 14:56:14.975510 4377 raft_consensus.cc:2781] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:56:14.975564 4377 raft_consensus.cc:482] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:56:14.975590 4377 raft_consensus.cc:3037] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:14.976612 4086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:14.970444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:44095 (local address 127.20.116.129:57011)
0928 14:56:14.970503 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:14.970518 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:14.970595 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:14.970789 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:14.970790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:14.970792 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:14.970871 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:14.970873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.972025 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.972026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:14.975425 (+ 3399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:14.975428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.976302 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.976303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:14.976389 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:14.976394 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:56:14.976464 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:56:14.976517 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 14:56:14.976660 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:14.970526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:57011 (local address 127.20.116.130:44095)
0928 14:56:14.970584 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:14.970586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:14.970596 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:56:14.970694 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:14.970696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:14.970708 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:14.970785 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:14.971224 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.971925 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.975526 (+ 3601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.976219 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.976294 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:14.976304 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:14.976496 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 14:56:14.976562 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:56:14.976657 4377 raft_consensus.cc:504] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:14.976822 4377 leader_election.cc:288] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [CANDIDATE]: Term 1 election: Requested vote from peers e7183525a8e942e390cc8b570a31e2fe (127.20.116.131:37981), 8f36604068874bc6a3eb9c379212b538 (127.20.116.130:44095)
I20260928 14:56:14.977049 4308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d790174725d54f93a875de8eb60bf53f" candidate_uuid: "42cafa23b6804de7a072f74483c2719c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7183525a8e942e390cc8b570a31e2fe"
I20260928 14:56:14.977128 4308 raft_consensus.cc:3037] T d790174725d54f93a875de8eb60bf53f P e7183525a8e942e390cc8b570a31e2fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:14.978184 4308 raft_consensus.cc:2454] T d790174725d54f93a875de8eb60bf53f P e7183525a8e942e390cc8b570a31e2fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cafa23b6804de7a072f74483c2719c in term 1.
I20260928 14:56:14.978343 3973 leader_election.cc:302] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cafa23b6804de7a072f74483c2719c, e7183525a8e942e390cc8b570a31e2fe; no voters:
I20260928 14:56:14.978441 4377 raft_consensus.cc:2781] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:14.978608 4377 raft_consensus.cc:686] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [term 1 LEADER]: Becoming Leader. State: Replica: 42cafa23b6804de7a072f74483c2719c, State: Running, Role: LEADER
I20260928 14:56:14.978713 4377 consensus_queue.cc:237] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:14.979055 4157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d790174725d54f93a875de8eb60bf53f" candidate_uuid: "42cafa23b6804de7a072f74483c2719c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f36604068874bc6a3eb9c379212b538" is_pre_election: true
I20260928 14:56:14.979070 4173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d790174725d54f93a875de8eb60bf53f" candidate_uuid: "42cafa23b6804de7a072f74483c2719c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f36604068874bc6a3eb9c379212b538"
I20260928 14:56:14.979202 4157 raft_consensus.cc:2454] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cafa23b6804de7a072f74483c2719c in term 0.
I20260928 14:56:14.979741 3892 catalog_manager.cc:5495] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cafa23b6804de7a072f74483c2719c (127.20.116.129). New cstate: current_term: 1 leader_uuid: "42cafa23b6804de7a072f74483c2719c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } health_report { overall_health: UNKNOWN } } }
I20260928 14:56:14.988637 4370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:14.986502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44762 (local address 127.20.116.129:35713)
0928 14:56:14.986565 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:14.986566 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:14.986575 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:56:14.986651 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:14.986652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:14.986668 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:14.986727 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:56:14.986866 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.987312 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.988033 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:14.988207 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:14.988264 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:14.988378 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:56:14.988471 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:56:14.988526 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:14.988558 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 14:56:14.988574 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:56:14.990348 4173 raft_consensus.cc:3037] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:14.990551 4308 raft_consensus.cc:1261] T d790174725d54f93a875de8eb60bf53f P e7183525a8e942e390cc8b570a31e2fe [term 1 FOLLOWER]: Refusing update from remote peer 42cafa23b6804de7a072f74483c2719c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:56:14.990895 4377 consensus_queue.cc:1035] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:56:14.991423 4173 raft_consensus.cc:1261] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538 [term 1 FOLLOWER]: Refusing update from remote peer 42cafa23b6804de7a072f74483c2719c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:56:14.991705 4377 consensus_queue.cc:1035] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:56:14.994217 4388 mvcc.cc:205] Tried to move back new op lower bound from 7334327807957770240 to 7334327807913406464. Current Snapshot: MvccSnapshot[applied={T|T < 7334327807957770240}]
I20260928 14:56:14.995963 3892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:56:14.996068 3892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:56:14.996789 3892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273: Sending DeleteTablet for 3 replicas of tablet d790174725d54f93a875de8eb60bf53f
I20260928 14:56:14.997129 4153 tablet_service.cc:1514] Processing DeleteTablet for tablet d790174725d54f93a875de8eb60bf53f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:14 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:35050
I20260928 14:56:14.997162 4288 tablet_service.cc:1514] Processing DeleteTablet for tablet d790174725d54f93a875de8eb60bf53f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:14 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:37328
I20260928 14:56:14.997356 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 3954
I20260928 14:56:14.997373 4017 tablet_service.cc:1514] Processing DeleteTablet for tablet d790174725d54f93a875de8eb60bf53f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:14 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:44752
W20260928 14:56:15.002816 3876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.116.129:35713 (error 108)
I20260928 14:56:15.002977 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 4089
I20260928 14:56:15.003353 4395 tablet_replica.cc:330] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538: stopping tablet replica
I20260928 14:56:15.003541 4396 tablet_replica.cc:330] T d790174725d54f93a875de8eb60bf53f P e7183525a8e942e390cc8b570a31e2fe: stopping tablet replica
I20260928 14:56:15.003640 4396 raft_consensus.cc:2229] T d790174725d54f93a875de8eb60bf53f P e7183525a8e942e390cc8b570a31e2fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:56:15.003631 4366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:15.003243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:35713 (local address 127.0.0.1:44770)
0928 14:56:15.003555 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:15.003601 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.129:35713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285}
I20260928 14:56:15.003760 4396 raft_consensus.cc:2258] T d790174725d54f93a875de8eb60bf53f P e7183525a8e942e390cc8b570a31e2fe [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 14:56:15.003770 3876 catalog_manager.cc:4584] TS 42cafa23b6804de7a072f74483c2719c (127.20.116.129:35713): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d790174725d54f93a875de8eb60bf53f: Network error: Client connection negotiation failed: client connection to 127.20.116.129:35713: connect: Connection refused (error 111)
I20260928 14:56:15.004298 4396 ts_tablet_manager.cc:1905] T d790174725d54f93a875de8eb60bf53f P e7183525a8e942e390cc8b570a31e2fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:15.006464 4396 ts_tablet_manager.cc:1918] T d790174725d54f93a875de8eb60bf53f P e7183525a8e942e390cc8b570a31e2fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:56:15.006579 4396 log.cc:1196] T d790174725d54f93a875de8eb60bf53f P e7183525a8e942e390cc8b570a31e2fe: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/d790174725d54f93a875de8eb60bf53f
I20260928 14:56:15.006888 4396 ts_tablet_manager.cc:1939] T d790174725d54f93a875de8eb60bf53f P e7183525a8e942e390cc8b570a31e2fe: Deleting consensus metadata
I20260928 14:56:15.007407 3879 catalog_manager.cc:4841] TS e7183525a8e942e390cc8b570a31e2fe (127.20.116.131:37981): tablet d790174725d54f93a875de8eb60bf53f (table test-table [id=dd84fe1d77c7439baceb42c707c9f894]) successfully deleted
I20260928 14:56:15.009034 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 4225
I20260928 14:56:15.009124 4366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:15.008972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:44095 (local address 127.0.0.1:35066)
0928 14:56:15.009035 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:15.009067 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.130:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 14:56:15.014273 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 3860
I20260928 14:56:35.019989 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:40571
--webserver_interface=127.20.116.190
--webserver_port=33869
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:40571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:35.105413 4398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:35.105679 4398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:35.105720 4398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:35.106974 4398 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 14:56:35.107043 4398 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 14:56:35.107066 4398 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 14:56:35.107084 4398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:56:35.107110 4398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:35.107143 4398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:35.107168 4398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:56:35.107195 4398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:56:35.108726 4398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:40571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:40571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=33869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:56:35.109011 4398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:35.109253 4398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:35.118955 4407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:35.118955 4404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:35.118997 4398 server_base.cc:1034] running on GCE node
W20260928 14:56:35.118955 4405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:35.119575 4398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:35.119948 4398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:35.121104 4398 hybrid_clock.cc:648] HybridClock initialized: now 1790607395121080 us; error 40 us; skew 500 ppm
Sep 28 14:56:35 dist-test-slave-pxl6 krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607395, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:35.136494 4398 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:56:35.137012 4398 webserver.cc:460] Webserver started at http://127.20.116.190:33869/ using document root <none> and password file <none>
I20260928 14:56:35.137290 4398 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:35.137351 4398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:35.138950 4398 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:56:35.139734 4414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:35.139935 4398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:56:35.140040 4398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "b0727c1267154abcaedf4f5d90513273"
format_stamp: "Formatted at 2026-09-28 14:56:14 on dist-test-slave-pxl6"
server_key: "c437ae2a158cec7faf9ab49f73bd0e4e"
server_key_iv: "d228d2c4bd100cf8b76daa5d0c37242a"
server_key_version: "encryptionkey@0"
I20260928 14:56:35.140476 4398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:35.152457 4398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:35.154631 4398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:35.154801 4398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:35.159143 4398 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:40571
I20260928 14:56:35.159183 4467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:40571 every 8 connection(s)
I20260928 14:56:35.159638 4398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:56:35.161226 4468 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:56:35.161948 4468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273: Bootstrap starting.
I20260928 14:56:35.165567 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 4398
I20260928 14:56:35.165880 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:35713
--local_ip_for_outbound_sockets=127.20.116.129
--tserver_master_addrs=127.20.116.190:40571
--webserver_port=34421
--webserver_interface=127.20.116.129
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:56:35.172456 4468 log.cc:824] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:35.174523 4468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:56:35.174913 4468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273: Bootstrap complete.
I20260928 14:56:35.177426 4468 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0727c1267154abcaedf4f5d90513273" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40571 } }
I20260928 14:56:35.177703 4468 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0727c1267154abcaedf4f5d90513273, State: Initialized, Role: FOLLOWER
I20260928 14:56:35.177899 4468 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0727c1267154abcaedf4f5d90513273" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40571 } }
I20260928 14:56:35.177979 4468 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:56:35.178040 4468 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:56:35.178099 4468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:56:35.179270 4468 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0727c1267154abcaedf4f5d90513273" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40571 } }
I20260928 14:56:35.179396 4468 leader_election.cc:302] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0727c1267154abcaedf4f5d90513273; no voters:
I20260928 14:56:35.179598 4468 leader_election.cc:288] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:56:35.179637 4472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:56:35.179844 4472 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 2 LEADER]: Becoming Leader. State: Replica: b0727c1267154abcaedf4f5d90513273, State: Running, Role: LEADER
I20260928 14:56:35.179924 4468 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:56:35.179962 4472 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0727c1267154abcaedf4f5d90513273" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40571 } }
I20260928 14:56:35.180243 4472 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0727c1267154abcaedf4f5d90513273. Latest consensus state: current_term: 2 leader_uuid: "b0727c1267154abcaedf4f5d90513273" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0727c1267154abcaedf4f5d90513273" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40571 } } }
I20260928 14:56:35.180325 4472 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:35.180248 4473 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b0727c1267154abcaedf4f5d90513273" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0727c1267154abcaedf4f5d90513273" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40571 } } }
I20260928 14:56:35.180492 4473 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:35.180542 4477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:56:35.181497 4477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:56:35.181633 4477 catalog_manager.cc:1259] Loaded cluster ID: c9d0ebc525e84a20b82a59d8ab138824
I20260928 14:56:35.181855 4477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:56:35.182862 4477 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:56:35.183122 4477 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273: Loaded TSK: 0
I20260928 14:56:35.193486 4477 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273: Generated new TSK 1
I20260928 14:56:35.193608 4477 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:56:35.262373 4470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:35.262638 4470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:35.262670 4470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:35.263903 4470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:35.263983 4470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:35.264012 4470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:56:35.265743 4470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:35713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=34421
--tserver_master_addrs=127.20.116.190:40571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:56:35.266002 4470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:35.266266 4470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:35.271618 4495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:35.271621 4496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:35.271919 4470 server_base.cc:1034] running on GCE node
W20260928 14:56:35.271621 4498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:35.272339 4470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:35.272645 4470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:35.273840 4470 hybrid_clock.cc:648] HybridClock initialized: now 1790607395273818 us; error 30 us; skew 500 ppm
Sep 28 14:56:35 dist-test-slave-pxl6 krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607395, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:35.276615 4470 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:56:35.277149 4470 webserver.cc:460] Webserver started at http://127.20.116.129:34421/ using document root <none> and password file <none>
I20260928 14:56:35.277397 4470 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:35.277511 4470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:35.279038 4470 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 14:56:35.279882 4505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:35.280102 4470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:56:35.280210 4470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "42cafa23b6804de7a072f74483c2719c"
format_stamp: "Formatted at 2026-09-28 14:56:14 on dist-test-slave-pxl6"
server_key: "c82fd90ba3a4cd311c37d65de7f438b9"
server_key_iv: "6f1fe9bcc9e14ccada892d2a6182d4e7"
server_key_version: "encryptionkey@0"
I20260928 14:56:35.280808 4470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:35.311975 4470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:35.314986 4470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:35.315160 4470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:35.315459 4470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:35.316071 4512 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 14:56:35.317800 4470 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:56:35.317864 4470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 14:56:35.317898 4470 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:56:35.318792 4470 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:56:35.318836 4470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260928 14:56:35.318941 4512 tablet_bootstrap.cc:492] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c: Bootstrap starting.
I20260928 14:56:35.325664 4470 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:35713
I20260928 14:56:35.325976 4620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:35713 every 8 connection(s)
I20260928 14:56:35.326128 4470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:56:35.331060 4512 log.cc:824] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:35.332284 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 4470
I20260928 14:56:35.332588 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:44095
--local_ip_for_outbound_sockets=127.20.116.130
--tserver_master_addrs=127.20.116.190:40571
--webserver_port=33645
--webserver_interface=127.20.116.130
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:56:35.334033 4512 tablet_bootstrap.cc:492] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:56:35.334338 4512 tablet_bootstrap.cc:492] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c: Bootstrap complete.
I20260928 14:56:35.335655 4512 ts_tablet_manager.cc:1397] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.012s
Sep 28 14:56:35 dist-test-slave-pxl6 krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607395, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:56:35.337072 4512 raft_consensus.cc:348] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:35.337404 4512 raft_consensus.cc:729] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cafa23b6804de7a072f74483c2719c, State: Initialized, Role: FOLLOWER
I20260928 14:56:35.337536 4512 consensus_queue.cc:260] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:35.337831 4512 ts_tablet_manager.cc:1428] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 14:56:35.341904 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.326989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.129:35667)
0928 14:56:35.327445 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:35.327507 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:35.328315 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:35.328865 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:35.328870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:35.329399 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:35.329767 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:35.329775 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.331551 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.331552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:35.332951 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:35.332956 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.333955 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.333957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:35.334145 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.334956 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:35.334969 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:35.336920 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:35.339850 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:35.339859 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:35.339861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:35.340162 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:35.340502 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:35.340503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:35.340504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:35.340643 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:35.341228 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:35.341236 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:35.341393 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 14:56:35.341509 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 14:56:35.341687 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":87,"threads_started":1}
I20260928 14:56:35.342499 4624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.327136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:35667 (local address 127.20.116.190:40571)
0928 14:56:35.327392 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:35.327403 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:35.327641 (+ 238us) server_negotiation.cc:407] Connection header received
0928 14:56:35.328574 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:35.328582 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:35.328645 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:35.328800 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:35.329912 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.331404 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.333077 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.333810 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.334704 (+ 894us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.337067 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:35.337072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:35.337073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:35.337088 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:35.339734 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:35.340266 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:35.340271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:35.340272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:35.340413 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:35.340729 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:35.340730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:35.340730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:35.341108 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:35.341247 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:35.341607 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 14:56:35.341793 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":138,"threads_started":1}
I20260928 14:56:35.343267 4621 heartbeater.cc:347] Connected to a master server at 127.20.116.190:40571
I20260928 14:56:35.343394 4621 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:35.343641 4621 heartbeater.cc:511] Master 127.20.116.190:40571 requested a full tablet report, sending...
I20260928 14:56:35.344635 4431 ts_manager.cc:194] Registered new tserver with Master: 42cafa23b6804de7a072f74483c2719c (127.20.116.129:35713)
I20260928 14:56:35.345592 4431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:35667
I20260928 14:56:35.346901 4621 heartbeater.cc:503] Master 127.20.116.190:40571 was elected leader, sending a full tablet report...
Sep 28 14:56:35 dist-test-slave-pxl6 krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607395, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for kudu/127.20.116.129@KRBTEST.COM
I20260928 14:56:35.355088 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.345362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:35713 (local address 127.0.0.1:49124)
0928 14:56:35.345844 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:35.345884 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:35.345964 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:35.347139 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:35.347146 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:35.347451 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:35.347508 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:35.347511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.348378 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.348379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:35.349029 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:35.349032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.349585 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.349586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:35.349676 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.350269 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:35.350274 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:35.351723 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:35.353689 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:35.353699 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:35.353702 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:35.353941 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:35.354195 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:35.354196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:35.354196 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:35.354265 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:35.354630 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:35.354641 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:35.354727 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 14:56:35.354829 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 14:56:35.354988 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":113,"threads_started":1}
I20260928 14:56:35.355088 4629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.345688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49124 (local address 127.20.116.129:35713)
0928 14:56:35.345892 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:35.346004 (+ 112us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:35.346021 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:56:35.346068 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:35.346071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:35.346096 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:35.347138 (+ 1042us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:35.347600 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.348283 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.349126 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.349503 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.349585 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.351832 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:35.351844 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:35.351845 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:35.351854 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:35.353584 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:35.354037 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:35.354043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:35.354043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:35.354110 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:35.354332 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:35.354333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:35.354334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:35.354498 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:35.354632 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:35.354871 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 14:56:35.354976 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":94,"threads_started":1}
I20260928 14:56:35.355818 4554 tablet_service.cc:1514] Processing DeleteTablet for tablet d790174725d54f93a875de8eb60bf53f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:14 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:49124
I20260928 14:56:35.356560 4630 tablet_replica.cc:330] stopping tablet replica
I20260928 14:56:35.356710 4630 raft_consensus.cc:2229] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:56:35.356803 4630 raft_consensus.cc:2258] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:35.357136 4630 ts_tablet_manager.cc:1905] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:35.359423 4630 ts_tablet_manager.cc:1918] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:56:35.359519 4630 log.cc:1196] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/wal/wals/d790174725d54f93a875de8eb60bf53f
I20260928 14:56:35.359776 4630 ts_tablet_manager.cc:1939] T d790174725d54f93a875de8eb60bf53f P 42cafa23b6804de7a072f74483c2719c: Deleting consensus metadata
I20260928 14:56:35.360358 4415 catalog_manager.cc:4841] TS 42cafa23b6804de7a072f74483c2719c (127.20.116.129:35713): tablet d790174725d54f93a875de8eb60bf53f (table test-table [id=dd84fe1d77c7439baceb42c707c9f894]) successfully deleted
W20260928 14:56:35.425936 4626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:35.426156 4626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:35.426182 4626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:35.427467 4626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:35.427510 4626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:35.427538 4626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:56:35.429028 4626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:44095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=33645
--tserver_master_addrs=127.20.116.190:40571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:56:35.429291 4626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:35.429550 4626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:35.434962 4636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:35.434957 4635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:35.434957 4638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:35.435204 4626 server_base.cc:1034] running on GCE node
I20260928 14:56:35.435432 4626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:35.435704 4626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:35.436875 4626 hybrid_clock.cc:648] HybridClock initialized: now 1790607395436849 us; error 40 us; skew 500 ppm
Sep 28 14:56:35 dist-test-slave-pxl6 krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607395, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:35.439788 4626 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:56:35.440313 4626 webserver.cc:460] Webserver started at http://127.20.116.130:33645/ using document root <none> and password file <none>
I20260928 14:56:35.440549 4626 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:35.440653 4626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:35.442160 4626 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:56:35.442953 4645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:35.443133 4626 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:56:35.443240 4626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "8f36604068874bc6a3eb9c379212b538"
format_stamp: "Formatted at 2026-09-28 14:56:14 on dist-test-slave-pxl6"
server_key: "cf5752ba887b8ec6c13f62a2763b3096"
server_key_iv: "e5017b0931bf38045b0ebcf9c25f833a"
server_key_version: "encryptionkey@0"
I20260928 14:56:35.443650 4626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:35.469646 4626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:35.472223 4626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:35.472396 4626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:35.472656 4626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:35.473107 4652 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 14:56:35.474781 4626 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:56:35.474958 4626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 14:56:35.475006 4626 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:56:35.475732 4626 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:56:35.475792 4626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 14:56:35.475920 4652 tablet_bootstrap.cc:492] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538: Bootstrap starting.
I20260928 14:56:35.483371 4626 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:44095
I20260928 14:56:35.483484 4759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:44095 every 8 connection(s)
I20260928 14:56:35.483860 4626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:56:35.486040 4652 log.cc:824] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:35.488611 4652 tablet_bootstrap.cc:492] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:56:35.488937 4652 tablet_bootstrap.cc:492] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538: Bootstrap complete.
I20260928 14:56:35.490156 4652 ts_tablet_manager.cc:1397] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.008s
Sep 28 14:56:35 dist-test-slave-pxl6 krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607395, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:56:35.491379 4652 raft_consensus.cc:348] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:35.491638 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 4626
I20260928 14:56:35.491655 4652 raft_consensus.cc:729] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f36604068874bc6a3eb9c379212b538, State: Initialized, Role: FOLLOWER
I20260928 14:56:35.491801 4652 consensus_queue.cc:260] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:35.491940 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:37981
--local_ip_for_outbound_sockets=127.20.116.131
--tserver_master_addrs=127.20.116.190:40571
--webserver_port=44487
--webserver_interface=127.20.116.131
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:56:35.492133 4652 ts_tablet_manager.cc:1428] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538: Time spent starting tablet: real 0.002s user 0.005s sys 0.001s
I20260928 14:56:35.495209 4624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.484647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:35823 (local address 127.20.116.190:40571)
0928 14:56:35.484710 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:35.484712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:35.485027 (+ 315us) server_negotiation.cc:407] Connection header received
0928 14:56:35.485809 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:35.485810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:35.485826 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:35.485883 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:35.486776 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.487287 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.488820 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.489099 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.489158 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.491820 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:35.491821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:35.491821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:35.491830 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:35.493645 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:35.494129 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:35.494131 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:35.494131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:35.494190 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:35.494462 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:35.494463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:35.494464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:35.494628 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:35.494737 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:35.495019 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 14:56:35.495116 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 14:56:35.495404 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.484593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.130:35823)
0928 14:56:35.484825 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:35.484897 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:35.485699 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:35.485967 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:35.485971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:35.486353 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:35.486666 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:35.486674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.487446 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.487447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:35.488736 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:35.488739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.489197 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.489199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:35.489354 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.489954 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:35.489973 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:35.491687 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:35.493749 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:35.493758 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:35.493760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:35.494045 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:35.494272 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:35.494273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:35.494274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:35.494390 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:35.494724 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:35.494732 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:35.494882 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 14:56:35.495000 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 14:56:35.495166 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":119,"threads_started":1}
I20260928 14:56:35.495916 4760 heartbeater.cc:347] Connected to a master server at 127.20.116.190:40571
I20260928 14:56:35.495998 4760 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:35.496206 4760 heartbeater.cc:511] Master 127.20.116.190:40571 requested a full tablet report, sending...
I20260928 14:56:35.496886 4431 ts_manager.cc:194] Registered new tserver with Master: 8f36604068874bc6a3eb9c379212b538 (127.20.116.130:44095)
I20260928 14:56:35.497589 4431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:35823
I20260928 14:56:35.498351 4760 heartbeater.cc:503] Master 127.20.116.190:40571 was elected leader, sending a full tablet report...
I20260928 14:56:35.502025 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.497759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59946 (local address 127.20.116.130:44095)
0928 14:56:35.498499 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:35.498503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:35.498513 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:56:35.498578 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:35.498580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:35.498596 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:35.498678 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:35.498853 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.499616 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.501200 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.501644 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.501707 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.501753 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:35.501943 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 14:56:35.501972 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":121,"threads_started":1}
I20260928 14:56:35.502104 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.497231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:44095 (local address 127.0.0.1:59946)
0928 14:56:35.497286 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:35.497296 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:35.497341 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:35.498706 (+ 1365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:35.498708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:35.498711 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:35.498771 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:35.498773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.499702 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.499704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:35.501103 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:35.501106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.501730 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.501731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:35.501819 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.501856 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 14:56:35.501928 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 14:56:35.501989 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 14:56:35.502508 4694 tablet_service.cc:1514] Processing DeleteTablet for tablet d790174725d54f93a875de8eb60bf53f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:14 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:59946
I20260928 14:56:35.503089 4767 tablet_replica.cc:330] stopping tablet replica
I20260928 14:56:35.503196 4767 raft_consensus.cc:2229] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:56:35.503268 4767 raft_consensus.cc:2258] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:35.503576 4767 ts_tablet_manager.cc:1905] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:35.505759 4767 ts_tablet_manager.cc:1918] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:56:35.505824 4767 log.cc:1196] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/d790174725d54f93a875de8eb60bf53f
I20260928 14:56:35.506045 4767 ts_tablet_manager.cc:1939] T d790174725d54f93a875de8eb60bf53f P 8f36604068874bc6a3eb9c379212b538: Deleting consensus metadata
I20260928 14:56:35.506613 4416 catalog_manager.cc:4841] TS 8f36604068874bc6a3eb9c379212b538 (127.20.116.130:44095): tablet d790174725d54f93a875de8eb60bf53f (table test-table [id=dd84fe1d77c7439baceb42c707c9f894]) successfully deleted
W20260928 14:56:35.580515 4765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:35.580713 4765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:35.580735 4765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:35.581918 4765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:35.581966 4765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:35.581993 4765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:56:35.583498 4765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:37981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=44487
--tserver_master_addrs=127.20.116.190:40571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:56:35.583743 4765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:35.583973 4765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:56:35.589125 4765 server_base.cc:1034] running on GCE node
W20260928 14:56:35.589068 4772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:35.589066 4775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:35.589056 4773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:35.589591 4765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:35.589859 4765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:35.591027 4765 hybrid_clock.cc:648] HybridClock initialized: now 1790607395590999 us; error 45 us; skew 500 ppm
Sep 28 14:56:35 dist-test-slave-pxl6 krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607395, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:35.593636 4765 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:56:35.594177 4765 webserver.cc:460] Webserver started at http://127.20.116.131:44487/ using document root <none> and password file <none>
I20260928 14:56:35.594429 4765 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:35.594522 4765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:35.596086 4765 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:56:35.596814 4782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:35.597049 4765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:56:35.597160 4765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "e7183525a8e942e390cc8b570a31e2fe"
format_stamp: "Formatted at 2026-09-28 14:56:14 on dist-test-slave-pxl6"
server_key: "c0b891636334f7684bbf6e9d4dbfadc0"
server_key_iv: "2fe9eda5f9ec2088f36eba87c361e6bb"
server_key_version: "encryptionkey@0"
I20260928 14:56:35.597568 4765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:35.616494 4765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:35.619024 4765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:35.619210 4765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:35.619470 4765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:35.619801 4765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:35.619860 4765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:35.619903 4765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:35.619952 4765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:35.625973 4765 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:37981
I20260928 14:56:35.626027 4895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:37981 every 8 connection(s)
I20260928 14:56:35.626428 4765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:56:35.628366 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 4765
Sep 28 14:56:35 dist-test-slave-pxl6 krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607395, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:56:35.636809 4900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.628825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49052 (local address 127.20.116.190:40571)
0928 14:56:35.628970 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:35.628973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:35.629067 (+ 94us) server_negotiation.cc:407] Connection header received
0928 14:56:35.629126 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:35.629127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:35.629138 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:35.629200 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:35.629789 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.630458 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.631677 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.631981 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.632063 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.633728 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:35.633730 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:35.633730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:35.633739 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:35.635461 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:35.635892 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:35.635893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:35.635894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:35.635973 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:35.636181 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:35.636182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:35.636182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:35.636336 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:35.636514 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:35.636554 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 14:56:35.636681 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260928 14:56:35.638240 4624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.627332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:46153 (local address 127.20.116.190:40571)
0928 14:56:35.627405 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:35.627407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:35.627642 (+ 235us) server_negotiation.cc:407] Connection header received
0928 14:56:35.628209 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:35.628210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:35.628224 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:35.628282 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:35.629622 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.630458 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.631934 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.632210 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.632259 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.635001 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:35.635002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:35.635003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:35.635011 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:35.636690 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:35.637179 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:35.637181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:35.637182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:35.637239 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:35.637534 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:35.637535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:35.637535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:35.637683 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:35.637761 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:35.638065 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 14:56:35.638145 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":44}
I20260928 14:56:35.638468 4898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.627318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.131:46153)
0928 14:56:35.627503 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:35.627541 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:35.628065 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:35.628369 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:35.628376 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:35.629324 (+ 948us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:35.629529 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:35.629534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.630619 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.630620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:35.631848 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:35.631851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.632299 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.632300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:35.632437 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.632889 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:35.632900 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:35.634910 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:35.636777 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:35.636785 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:35.636788 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:35.637099 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:35.637317 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:35.637318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:35.637318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:35.637443 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:35.637767 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:35.637775 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:35.637922 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 14:56:35.638032 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 14:56:35.638216 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":83,"threads_started":1}
I20260928 14:56:35.639019 4896 heartbeater.cc:347] Connected to a master server at 127.20.116.190:40571
I20260928 14:56:35.639161 4896 heartbeater.cc:464] Registering TS with master...
I20260928 14:56:35.639361 4896 heartbeater.cc:511] Master 127.20.116.190:40571 requested a full tablet report, sending...
I20260928 14:56:35.639815 4431 ts_manager.cc:194] Registered new tserver with Master: e7183525a8e942e390cc8b570a31e2fe (127.20.116.131:37981)
I20260928 14:56:35.640187 4431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:46153
I20260928 14:56:35.648523 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:56:35.651767 4624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.648948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49064 (local address 127.20.116.190:40571)
0928 14:56:35.649048 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:35.649051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:35.649166 (+ 115us) server_negotiation.cc:407] Connection header received
0928 14:56:35.649274 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:35.649274 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:35.649310 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:35.649394 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:56:35.649545 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.650039 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.650942 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.651192 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.651274 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.651460 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:56:35.651505 (+ 45us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 14:56:35.651681 (+ 176us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49064: token expired
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 14:56:35.651844 4624 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49064: token expired
W20260928 14:56:35.651913 4359 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.20.116.190:40571)
I20260928 14:56:35.657985 4624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.652170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49080 (local address 127.20.116.190:40571)
0928 14:56:35.652206 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:35.652208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:35.652218 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:56:35.652282 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:35.652282 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:35.652294 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:35.652329 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:35.652831 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.653272 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.653979 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.654162 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.654215 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.655504 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:35.655505 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:35.655505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:35.655512 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:35.657034 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:35.657347 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:35.657348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:35.657348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:35.657403 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:35.657568 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:35.657569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:35.657569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:35.657702 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:35.657779 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:35.657864 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 14:56:35.657923 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 14:56:35.659246 4431 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:56:35.660923 4431 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:56:35.663250 4694 tablet_service.cc:1467] Processing CreateTablet for tablet b097fc4f95904ea1a6898da170a9814a (DEFAULT_TABLE table=test-table [id=06aab6d83d0441c099545a2a9310cf13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:35.663324 4554 tablet_service.cc:1467] Processing CreateTablet for tablet b097fc4f95904ea1a6898da170a9814a (DEFAULT_TABLE table=test-table [id=06aab6d83d0441c099545a2a9310cf13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:35.663461 4694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b097fc4f95904ea1a6898da170a9814a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:35.663546 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b097fc4f95904ea1a6898da170a9814a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:35.664713 4652 tablet_bootstrap.cc:492] T b097fc4f95904ea1a6898da170a9814a P 8f36604068874bc6a3eb9c379212b538: Bootstrap starting.
I20260928 14:56:35.665028 4652 tablet_bootstrap.cc:654] T b097fc4f95904ea1a6898da170a9814a P 8f36604068874bc6a3eb9c379212b538: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:35.665196 4512 tablet_bootstrap.cc:492] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c: Bootstrap starting.
I20260928 14:56:35.665486 4512 tablet_bootstrap.cc:654] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:35.665596 4652 tablet_bootstrap.cc:492] T b097fc4f95904ea1a6898da170a9814a P 8f36604068874bc6a3eb9c379212b538: No bootstrap required, opened a new log
I20260928 14:56:35.665645 4652 ts_tablet_manager.cc:1397] T b097fc4f95904ea1a6898da170a9814a P 8f36604068874bc6a3eb9c379212b538: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:56:35.665759 4652 raft_consensus.cc:348] T b097fc4f95904ea1a6898da170a9814a P 8f36604068874bc6a3eb9c379212b538 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:35.665832 4652 raft_consensus.cc:374] T b097fc4f95904ea1a6898da170a9814a P 8f36604068874bc6a3eb9c379212b538 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:35.665884 4652 raft_consensus.cc:729] T b097fc4f95904ea1a6898da170a9814a P 8f36604068874bc6a3eb9c379212b538 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f36604068874bc6a3eb9c379212b538, State: Initialized, Role: FOLLOWER
I20260928 14:56:35.665931 4652 consensus_queue.cc:260] T b097fc4f95904ea1a6898da170a9814a P 8f36604068874bc6a3eb9c379212b538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:35.666021 4652 ts_tablet_manager.cc:1428] T b097fc4f95904ea1a6898da170a9814a P 8f36604068874bc6a3eb9c379212b538: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:35.666131 4512 tablet_bootstrap.cc:492] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c: No bootstrap required, opened a new log
I20260928 14:56:35.666188 4512 ts_tablet_manager.cc:1397] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:56:35.666601 4512 raft_consensus.cc:348] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:35.666682 4512 raft_consensus.cc:374] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:35.666723 4512 raft_consensus.cc:729] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cafa23b6804de7a072f74483c2719c, State: Initialized, Role: FOLLOWER
I20260928 14:56:35.666770 4512 consensus_queue.cc:260] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:35.666860 4512 ts_tablet_manager.cc:1428] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:56:35.667493 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.663037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:37981 (local address 127.0.0.1:53288)
0928 14:56:35.663077 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:35.663094 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:35.663137 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:35.663736 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:35.663737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:35.663739 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:35.663783 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:35.663784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.665057 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.665058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:35.666171 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:35.666174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.667212 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.667213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:35.667278 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.667282 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:56:35.667370 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 14:56:35.667417 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 14:56:35.667508 4902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.663227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53288 (local address 127.20.116.131:37981)
0928 14:56:35.663472 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:35.663477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:35.663488 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:56:35.663531 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:35.663533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:35.663556 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:35.663769 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:35.663854 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.664950 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.666283 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.667106 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.667181 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.667244 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:35.667390 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 14:56:35.667430 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":119,"threads_started":1}
I20260928 14:56:35.668246 4830 tablet_service.cc:1467] Processing CreateTablet for tablet b097fc4f95904ea1a6898da170a9814a (DEFAULT_TABLE table=test-table [id=06aab6d83d0441c099545a2a9310cf13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:56:35.668550 4830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b097fc4f95904ea1a6898da170a9814a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:56:35.671691 4905 tablet_bootstrap.cc:492] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe: Bootstrap starting.
I20260928 14:56:35.672324 4905 tablet_bootstrap.cc:654] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe: Neither blocks nor log segments found. Creating new log.
I20260928 14:56:35.672950 4905 log.cc:824] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:35.673754 4905 tablet_bootstrap.cc:492] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe: No bootstrap required, opened a new log
I20260928 14:56:35.673818 4905 ts_tablet_manager.cc:1397] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:56:35.675299 4905 raft_consensus.cc:348] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:35.675449 4905 raft_consensus.cc:374] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:56:35.675472 4905 raft_consensus.cc:729] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7183525a8e942e390cc8b570a31e2fe, State: Initialized, Role: FOLLOWER
I20260928 14:56:35.675611 4905 consensus_queue.cc:260] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:35.675904 4905 ts_tablet_manager.cc:1428] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:56:35.675964 4896 heartbeater.cc:503] Master 127.20.116.190:40571 was elected leader, sending a full tablet report...
W20260928 14:56:35.734778 4761 tablet.cc:2369] T b097fc4f95904ea1a6898da170a9814a P 8f36604068874bc6a3eb9c379212b538: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:56:35.739322 4907 raft_consensus.cc:482] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:56:35.739442 4907 raft_consensus.cc:504] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:35.739898 4907 leader_election.cc:288] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cafa23b6804de7a072f74483c2719c (127.20.116.129:35713), 8f36604068874bc6a3eb9c379212b538 (127.20.116.130:44095)
I20260928 14:56:35.743981 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.740169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:42417 (local address 127.20.116.130:44095)
0928 14:56:35.740213 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:35.740216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:35.740409 (+ 193us) server_negotiation.cc:407] Connection header received
0928 14:56:35.740444 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:35.740445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:35.740454 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:35.740508 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:35.740680 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.741421 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.742988 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.743577 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.743688 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.743700 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:35.743866 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 14:56:35.743912 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:56:35.743991 4629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.740130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:40637 (local address 127.20.116.129:35713)
0928 14:56:35.740182 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:35.740185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:35.740196 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:56:35.740271 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:35.740272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:35.740298 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:35.740367 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:56:35.740578 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.741419 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.742988 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.743700 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.743771 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.743859 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:35.743884 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 14:56:35.743931 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 14:56:35.743983 4898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.740074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:35713 (local address 127.20.116.131:40637)
0928 14:56:35.740134 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:35.740146 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:35.740189 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:35.740394 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:35.740395 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:35.740397 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:35.740478 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:35.740480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.741535 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.741536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:35.742872 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:35.742876 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.743760 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.743761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:35.743835 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.743838 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:56:35.743880 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 14:56:35.743907 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":46}
I20260928 14:56:35.743991 4908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.740083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:44095 (local address 127.20.116.131:42417)
0928 14:56:35.740314 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:35.740322 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:35.740392 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:35.740534 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:35.740535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:35.740537 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:56:35.740598 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:35.740599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.741535 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.741536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:35.742877 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:35.742878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.743685 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.743686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:35.743778 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.743783 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:56:35.743844 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 14:56:35.743893 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":116,"thread_start_us":105,"threads_started":1}
I20260928 14:56:35.744378 4714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b097fc4f95904ea1a6898da170a9814a" candidate_uuid: "e7183525a8e942e390cc8b570a31e2fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f36604068874bc6a3eb9c379212b538" is_pre_election: true
I20260928 14:56:35.744370 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b097fc4f95904ea1a6898da170a9814a" candidate_uuid: "e7183525a8e942e390cc8b570a31e2fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cafa23b6804de7a072f74483c2719c" is_pre_election: true
I20260928 14:56:35.744541 4714 raft_consensus.cc:2454] T b097fc4f95904ea1a6898da170a9814a P 8f36604068874bc6a3eb9c379212b538 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7183525a8e942e390cc8b570a31e2fe in term 0.
I20260928 14:56:35.744540 4574 raft_consensus.cc:2454] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7183525a8e942e390cc8b570a31e2fe in term 0.
I20260928 14:56:35.744776 4784 leader_election.cc:302] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f36604068874bc6a3eb9c379212b538, e7183525a8e942e390cc8b570a31e2fe; no voters:
I20260928 14:56:35.744943 4907 raft_consensus.cc:2781] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:56:35.745007 4907 raft_consensus.cc:482] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:56:35.745024 4907 raft_consensus.cc:3037] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:35.745774 4907 raft_consensus.cc:504] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:35.745929 4907 leader_election.cc:288] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe [CANDIDATE]: Term 1 election: Requested vote from peers 42cafa23b6804de7a072f74483c2719c (127.20.116.129:35713), 8f36604068874bc6a3eb9c379212b538 (127.20.116.130:44095)
I20260928 14:56:35.746076 4714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b097fc4f95904ea1a6898da170a9814a" candidate_uuid: "e7183525a8e942e390cc8b570a31e2fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f36604068874bc6a3eb9c379212b538"
I20260928 14:56:35.746166 4714 raft_consensus.cc:3037] T b097fc4f95904ea1a6898da170a9814a P 8f36604068874bc6a3eb9c379212b538 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:35.747043 4714 raft_consensus.cc:2454] T b097fc4f95904ea1a6898da170a9814a P 8f36604068874bc6a3eb9c379212b538 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7183525a8e942e390cc8b570a31e2fe in term 1.
I20260928 14:56:35.747143 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b097fc4f95904ea1a6898da170a9814a" candidate_uuid: "e7183525a8e942e390cc8b570a31e2fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cafa23b6804de7a072f74483c2719c"
I20260928 14:56:35.747244 4784 leader_election.cc:302] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f36604068874bc6a3eb9c379212b538, e7183525a8e942e390cc8b570a31e2fe; no voters:
I20260928 14:56:35.747257 4574 raft_consensus.cc:3037] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:56:35.747344 4907 raft_consensus.cc:2781] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:56:35.747519 4907 raft_consensus.cc:686] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe [term 1 LEADER]: Becoming Leader. State: Replica: e7183525a8e942e390cc8b570a31e2fe, State: Running, Role: LEADER
I20260928 14:56:35.747694 4907 consensus_queue.cc:237] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:35.748128 4574 raft_consensus.cc:2454] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7183525a8e942e390cc8b570a31e2fe in term 1.
I20260928 14:56:35.748636 4431 catalog_manager.cc:5495] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe reported cstate change: term changed from 0 to 1, leader changed from <none> to e7183525a8e942e390cc8b570a31e2fe (127.20.116.131). New cstate: current_term: 1 leader_uuid: "e7183525a8e942e390cc8b570a31e2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } health_report { overall_health: UNKNOWN } } }
I20260928 14:56:35.761518 4902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.759284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.20.116.131:37981)
0928 14:56:35.759328 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:35.759330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:35.759338 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:56:35.759416 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:35.759417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:35.759435 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:35.759490 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:56:35.759604 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.760064 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.760758 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:35.760945 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:35.761003 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:35.761194 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:56:35.761294 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:56:35.761358 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:35.761404 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 14:56:35.761422 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:56:35.763516 4574 raft_consensus.cc:1261] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Refusing update from remote peer e7183525a8e942e390cc8b570a31e2fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:56:35.763516 4714 raft_consensus.cc:1261] T b097fc4f95904ea1a6898da170a9814a P 8f36604068874bc6a3eb9c379212b538 [term 1 FOLLOWER]: Refusing update from remote peer e7183525a8e942e390cc8b570a31e2fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:56:35.763912 4909 consensus_queue.cc:1035] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:56:35.764045 4907 consensus_queue.cc:1035] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:56:35.766515 4912 mvcc.cc:205] Tried to move back new op lower bound from 7334327893044375552 to 7334327892983300096. Current Snapshot: MvccSnapshot[applied={T|T < 7334327893044375552}]
I20260928 14:56:35.769542 4431 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:56:35.769651 4431 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:56:35.770413 4431 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273: Sending DeleteTablet for 3 replicas of tablet b097fc4f95904ea1a6898da170a9814a
I20260928 14:56:35.770736 4554 tablet_service.cc:1514] Processing DeleteTablet for tablet b097fc4f95904ea1a6898da170a9814a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:35 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:49124
I20260928 14:56:35.770769 4694 tablet_service.cc:1514] Processing DeleteTablet for tablet b097fc4f95904ea1a6898da170a9814a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:35 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:59946
I20260928 14:56:35.770843 4630 tablet_replica.cc:330] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c: stopping tablet replica
I20260928 14:56:35.770896 4630 raft_consensus.cc:2229] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:56:35.770946 4630 raft_consensus.cc:2258] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:35.770922 4830 tablet_service.cc:1514] Processing DeleteTablet for tablet b097fc4f95904ea1a6898da170a9814a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:56:35 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:53288
I20260928 14:56:35.771034 4767 tablet_replica.cc:330] T b097fc4f95904ea1a6898da170a9814a P 8f36604068874bc6a3eb9c379212b538: stopping tablet replica
I20260928 14:56:35.771095 4767 raft_consensus.cc:2229] T b097fc4f95904ea1a6898da170a9814a P 8f36604068874bc6a3eb9c379212b538 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:56:35.771135 4767 raft_consensus.cc:2258] T b097fc4f95904ea1a6898da170a9814a P 8f36604068874bc6a3eb9c379212b538 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:35.771360 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 4470
I20260928 14:56:35.771526 4767 ts_tablet_manager.cc:1905] T b097fc4f95904ea1a6898da170a9814a P 8f36604068874bc6a3eb9c379212b538: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:35.773581 4767 ts_tablet_manager.cc:1918] T b097fc4f95904ea1a6898da170a9814a P 8f36604068874bc6a3eb9c379212b538: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:56:35.773639 4767 log.cc:1196] T b097fc4f95904ea1a6898da170a9814a P 8f36604068874bc6a3eb9c379212b538: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/b097fc4f95904ea1a6898da170a9814a
I20260928 14:56:35.773864 4767 ts_tablet_manager.cc:1939] T b097fc4f95904ea1a6898da170a9814a P 8f36604068874bc6a3eb9c379212b538: Deleting consensus metadata
I20260928 14:56:35.774279 4416 catalog_manager.cc:4841] TS 8f36604068874bc6a3eb9c379212b538 (127.20.116.130:44095): tablet b097fc4f95904ea1a6898da170a9814a (table test-table [id=06aab6d83d0441c099545a2a9310cf13]) successfully deleted
I20260928 14:56:35.774667 4922 tablet_replica.cc:330] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe: stopping tablet replica
I20260928 14:56:35.774803 4922 raft_consensus.cc:2229] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:56:35.774924 4922 raft_consensus.cc:2258] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:56:35.775432 4922 ts_tablet_manager.cc:1905] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:56:35.777423 4922 ts_tablet_manager.cc:1918] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:56:35.777509 4922 log.cc:1196] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/b097fc4f95904ea1a6898da170a9814a
I20260928 14:56:35.777804 4922 ts_tablet_manager.cc:1939] T b097fc4f95904ea1a6898da170a9814a P e7183525a8e942e390cc8b570a31e2fe: Deleting consensus metadata
I20260928 14:56:35.778311 4418 catalog_manager.cc:4841] TS e7183525a8e942e390cc8b570a31e2fe (127.20.116.131:37981): tablet b097fc4f95904ea1a6898da170a9814a (table test-table [id=06aab6d83d0441c099545a2a9310cf13]) successfully deleted
W20260928 14:56:35.779361 4415 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.116.129:35713 (error 108)
I20260928 14:56:35.779649 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 4626
I20260928 14:56:35.780344 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:35.779776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:35713 (local address 127.0.0.1:49132)
0928 14:56:35.780254 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:35.780302 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.129:35713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456}
W20260928 14:56:35.780499 4415 catalog_manager.cc:4584] TS 42cafa23b6804de7a072f74483c2719c (127.20.116.129:35713): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b097fc4f95904ea1a6898da170a9814a: Network error: Client connection negotiation failed: client connection to 127.20.116.129:35713: connect: Connection refused (error 111)
I20260928 14:56:35.785918 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 4765
I20260928 14:56:35.791330 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 4398
I20260928 14:56:55.797529 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:40571
--webserver_interface=127.20.116.190
--webserver_port=33869
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:40571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:55.882465 4924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:55.882747 4924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:55.882777 4924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:55.883929 4924 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 14:56:55.883965 4924 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 14:56:55.883977 4924 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 14:56:55.883988 4924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:56:55.884002 4924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:55.884014 4924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:55.884027 4924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:56:55.884038 4924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:56:55.885496 4924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:40571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:40571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=33869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:56:55.885695 4924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:55.885869 4924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:56:55.891079 4924 server_base.cc:1034] running on GCE node
W20260928 14:56:55.891067 4929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:55.891180 4932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:55.891088 4930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:55.891491 4924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:55.891803 4924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:55.892980 4924 hybrid_clock.cc:648] HybridClock initialized: now 1790607415892948 us; error 48 us; skew 500 ppm
Sep 28 14:56:55 dist-test-slave-pxl6 krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607415, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:55.903926 4924 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:56:55.904453 4924 webserver.cc:460] Webserver started at http://127.20.116.190:33869/ using document root <none> and password file <none>
I20260928 14:56:55.904711 4924 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:55.904786 4924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:55.906251 4924 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:56:55.907152 4939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:55.907373 4924 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:56:55.907490 4924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "b0727c1267154abcaedf4f5d90513273"
format_stamp: "Formatted at 2026-09-28 14:56:14 on dist-test-slave-pxl6"
server_key: "c437ae2a158cec7faf9ab49f73bd0e4e"
server_key_iv: "d228d2c4bd100cf8b76daa5d0c37242a"
server_key_version: "encryptionkey@0"
I20260928 14:56:55.907883 4924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:55.929800 4924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:55.932421 4924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:55.932605 4924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:55.937268 4924 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:40571
I20260928 14:56:55.937316 4991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:40571 every 8 connection(s)
I20260928 14:56:55.937731 4924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:56:55.939451 4992 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:56:55.940109 4992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273: Bootstrap starting.
I20260928 14:56:55.943064 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 4924
I20260928 14:56:55.943362 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:35713
--local_ip_for_outbound_sockets=127.20.116.129
--tserver_master_addrs=127.20.116.190:40571
--webserver_port=34421
--webserver_interface=127.20.116.129
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:56:55.949455 4992 log.cc:824] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:55.951972 4992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:56:55.952317 4992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273: Bootstrap complete.
I20260928 14:56:55.954752 4992 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0727c1267154abcaedf4f5d90513273" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40571 } }
I20260928 14:56:55.954991 4992 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0727c1267154abcaedf4f5d90513273, State: Initialized, Role: FOLLOWER
I20260928 14:56:55.955178 4992 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0727c1267154abcaedf4f5d90513273" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40571 } }
I20260928 14:56:55.955242 4992 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:56:55.955298 4992 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:56:55.955348 4992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 2 FOLLOWER]: Advancing to term 3
I20260928 14:56:55.956370 4992 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0727c1267154abcaedf4f5d90513273" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40571 } }
I20260928 14:56:55.956496 4992 leader_election.cc:302] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0727c1267154abcaedf4f5d90513273; no voters:
I20260928 14:56:55.956759 4992 leader_election.cc:288] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 14:56:55.956800 4996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 14:56:55.957005 4996 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [term 3 LEADER]: Becoming Leader. State: Replica: b0727c1267154abcaedf4f5d90513273, State: Running, Role: LEADER
I20260928 14:56:55.957062 4992 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:56:55.957101 4996 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0727c1267154abcaedf4f5d90513273" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40571 } }
I20260928 14:56:55.957408 4996 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0727c1267154abcaedf4f5d90513273. Latest consensus state: current_term: 3 leader_uuid: "b0727c1267154abcaedf4f5d90513273" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0727c1267154abcaedf4f5d90513273" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40571 } } }
I20260928 14:56:55.957682 4996 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:55.957408 4998 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b0727c1267154abcaedf4f5d90513273" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0727c1267154abcaedf4f5d90513273" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40571 } } }
I20260928 14:56:55.957971 4998 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273 [sys.catalog]: This master's current role is: LEADER
I20260928 14:56:55.958078 5006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:56:55.958973 5006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:56:55.959074 5006 catalog_manager.cc:1259] Loaded cluster ID: c9d0ebc525e84a20b82a59d8ab138824
I20260928 14:56:55.959112 5006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:56:55.960336 5006 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:56:55.960522 5006 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273: Loaded TSK: 0
I20260928 14:56:55.960573 5006 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b0727c1267154abcaedf4f5d90513273: Loaded TSK: 1
F20260928 14:56:55.960803 5006 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff67643a9ad google::LogMessage::Fail() at ??:0
@ 0x7ff67643d88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff67643a4de google::LogMessage::Flush() at ??:0
@ 0x7ff67643ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff678ba7104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff678ba4d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff67e3941cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff67e39448e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff67e394594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7ff67e35ac48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ff67e364ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff6783080d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff6782fd954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff6791616db start_thread at ??:0
@ 0x7ff674f3861f clone at ??:0
W20260928 14:56:56.041193 4994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:56.041460 4994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:56.041503 4994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:56.042819 4994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:56.042888 4994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:56.042936 4994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:56:56.044512 4994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:35713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=34421
--tserver_master_addrs=127.20.116.190:40571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:56:56.044818 4994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:56.045081 4994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:56.050733 5034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:56.050738 5033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:56.050768 5036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:56.051494 4994 server_base.cc:1034] running on GCE node
I20260928 14:56:56.051815 4994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:56.052115 4994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:56.053354 4994 hybrid_clock.cc:648] HybridClock initialized: now 1790607416053335 us; error 35 us; skew 500 ppm
Sep 28 14:56:56 dist-test-slave-pxl6 krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607416, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:56.056291 4994 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:56:56.056879 4994 webserver.cc:460] Webserver started at http://127.20.116.129:34421/ using document root <none> and password file <none>
I20260928 14:56:56.057152 4994 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:56.057227 4994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:56.058760 4994 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:56:56.059573 5044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:56.059818 4994 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:56:56.059924 4994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "42cafa23b6804de7a072f74483c2719c"
format_stamp: "Formatted at 2026-09-28 14:56:14 on dist-test-slave-pxl6"
server_key: "c82fd90ba3a4cd311c37d65de7f438b9"
server_key_iv: "6f1fe9bcc9e14ccada892d2a6182d4e7"
server_key_version: "encryptionkey@0"
I20260928 14:56:56.060339 4994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:56.069386 4994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:56.071913 4994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:56.072086 4994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:56.072348 4994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:56.072844 5051 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 14:56:56.074965 4994 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:56:56.075059 4994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 14:56:56.075100 4994 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:56:56.075824 4994 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:56:56.075891 4994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 14:56:56.075964 5051 tablet_bootstrap.cc:492] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c: Bootstrap starting.
I20260928 14:56:56.083403 4994 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:35713
I20260928 14:56:56.083556 5158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:35713 every 8 connection(s)
I20260928 14:56:56.083869 4994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:56:56.087634 5051 log.cc:824] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:56:56.088884 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 4994
I20260928 14:56:56.089288 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:44095
--local_ip_for_outbound_sockets=127.20.116.130
--tserver_master_addrs=127.20.116.190:40571
--webserver_port=33645
--webserver_interface=127.20.116.130
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:56:56.090673 5051 tablet_bootstrap.cc:492] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:56:56.091049 5051 tablet_bootstrap.cc:492] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c: Bootstrap complete.
I20260928 14:56:56.092360 5051 ts_tablet_manager.cc:1397] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.004s
I20260928 14:56:56.093914 5051 raft_consensus.cc:348] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:56.094414 5051 raft_consensus.cc:729] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cafa23b6804de7a072f74483c2719c, State: Initialized, Role: FOLLOWER
I20260928 14:56:56.094678 5051 consensus_queue.cc:260] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:56.094980 5051 ts_tablet_manager.cc:1428] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260928 14:56:56.175477 5161 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:56:56.084595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.129:60043)
0928 14:56:56.084805 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:56.084853 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:56.085362 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:56.175267 (+ 89905us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":133,"threads_started":1}
W20260928 14:56:56.175799 5159 heartbeater.cc:644] Failed to heartbeat to 127.20.116.190:40571 (0 consecutive failures): Network error: Failed to ping master at 127.20.116.190:40571: Client connection negotiation failed: client connection to 127.20.116.190:40571: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:56:56.176214 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:56.176075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.129:40325)
0928 14:56:56.176135 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:56.176186 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 14:56:56.176669 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:56.176511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.129:46157)
0928 14:56:56.176563 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:56.176649 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 14:56:56.176819 5159 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:56:56.177112 5163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:56.177302 5163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:56.177354 5163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:56.178608 5163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:56.178658 5163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:56.178686 5163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:56:56.180227 5163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:44095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=33645
--tserver_master_addrs=127.20.116.190:40571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:56:56.180485 5163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:56.180734 5163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:56.186057 5172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:56.186057 5170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:56.186299 5169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:56.186355 5163 server_base.cc:1034] running on GCE node
I20260928 14:56:56.186671 5163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:56.186985 5163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:56.188172 5163 hybrid_clock.cc:648] HybridClock initialized: now 1790607416188162 us; error 49 us; skew 500 ppm
Sep 28 14:56:56 dist-test-slave-pxl6 krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607416, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:56.191038 5163 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:56:56.191553 5163 webserver.cc:460] Webserver started at http://127.20.116.130:33645/ using document root <none> and password file <none>
I20260928 14:56:56.191804 5163 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:56.191907 5163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:56.193851 5163 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:56:56.194656 5179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:56.194869 5163 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:56:56.194976 5163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "8f36604068874bc6a3eb9c379212b538"
format_stamp: "Formatted at 2026-09-28 14:56:14 on dist-test-slave-pxl6"
server_key: "cf5752ba887b8ec6c13f62a2763b3096"
server_key_iv: "e5017b0931bf38045b0ebcf9c25f833a"
server_key_version: "encryptionkey@0"
I20260928 14:56:56.195364 5163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:56.209553 5163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:56.212129 5163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:56.212309 5163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:56.212565 5163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:56.212921 5163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:56.212989 5163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:56.213030 5163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:56.213059 5163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:56.219444 5163 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:44095
I20260928 14:56:56.219488 5292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:44095 every 8 connection(s)
I20260928 14:56:56.219872 5163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:56:56.220958 5295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:56.220596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.130:38835)
0928 14:56:56.220749 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:56.220820 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
W20260928 14:56:56.221385 5293 heartbeater.cc:644] Failed to heartbeat to 127.20.116.190:40571 (0 consecutive failures): Network error: Failed to ping master at 127.20.116.190:40571: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
I20260928 14:56:56.221662 5295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:56.221571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.130:60619)
0928 14:56:56.221618 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:56.221644 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 14:56:56.221956 5295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:56.221900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.130:43523)
0928 14:56:56.221922 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:56.221942 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 14:56:56.222066 5293 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:56:56.225746 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 5163
I20260928 14:56:56.225996 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:37981
--local_ip_for_outbound_sockets=127.20.116.131
--tserver_master_addrs=127.20.116.190:40571
--webserver_port=44487
--webserver_interface=127.20.116.131
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:56:56.310025 5296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:56:56.310259 5296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:56:56.310292 5296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:56:56.311686 5296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:56:56.311766 5296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:56:56.311797 5296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:56:56.313620 5296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:37981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=44487
--tserver_master_addrs=127.20.116.190:40571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:56:56.313853 5296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:56:56.314123 5296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:56:56.319433 5304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:56.319433 5301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:56:56.319449 5302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:56:56.319741 5296 server_base.cc:1034] running on GCE node
I20260928 14:56:56.320004 5296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:56:56.320286 5296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:56:56.321506 5296 hybrid_clock.cc:648] HybridClock initialized: now 1790607416321491 us; error 30 us; skew 500 ppm
Sep 28 14:56:56 dist-test-slave-pxl6 krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607416, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:56:56.324692 5296 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:56:56.325136 5296 webserver.cc:460] Webserver started at http://127.20.116.131:44487/ using document root <none> and password file <none>
I20260928 14:56:56.325368 5296 fs_manager.cc:359] Metadata directory not provided
I20260928 14:56:56.325415 5296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:56:56.326915 5296 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:56:56.327540 5311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:56.327749 5296 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:56:56.327854 5296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "e7183525a8e942e390cc8b570a31e2fe"
format_stamp: "Formatted at 2026-09-28 14:56:14 on dist-test-slave-pxl6"
server_key: "c0b891636334f7684bbf6e9d4dbfadc0"
server_key_iv: "2fe9eda5f9ec2088f36eba87c361e6bb"
server_key_version: "encryptionkey@0"
I20260928 14:56:56.328229 5296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:56:56.356413 5296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:56:56.359047 5296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:56:56.359186 5296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:56:56.359503 5296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:56:56.359855 5296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:56:56.359891 5296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:56.359915 5296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:56:56.359965 5296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:56:56.366112 5296 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:37981
I20260928 14:56:56.366223 5424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:37981 every 8 connection(s)
I20260928 14:56:56.366598 5296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:56:56.367810 5427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:56.367413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.131:45483)
0928 14:56:56.367584 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:56.367661 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
W20260928 14:56:56.368281 5425 heartbeater.cc:644] Failed to heartbeat to 127.20.116.190:40571 (0 consecutive failures): Network error: Failed to ping master at 127.20.116.190:40571: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
I20260928 14:56:56.368692 5427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:56.368514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.131:53803)
0928 14:56:56.368592 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:56.368670 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 14:56:56.369037 5427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:56.368977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.131:47403)
0928 14:56:56.369002 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:56.369023 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260928 14:56:56.369201 5425 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:56:56.371521 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 5296
I20260928 14:56:57.177591 5429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:57.177285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.129:55343)
0928 14:56:57.177463 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:57.177513 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":87,"threads_started":1}
I20260928 14:56:57.222645 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:57.222381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.130:41323)
0928 14:56:57.222508 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:57.222596 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":58,"threads_started":1}
I20260928 14:56:57.369907 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:57.369601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.131:47933)
0928 14:56:57.369806 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:57.369845 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":112,"threads_started":1}
I20260928 14:56:57.737421 5432 raft_consensus.cc:482] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:56:57.737604 5432 raft_consensus.cc:504] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:57.738008 5432 leader_election.cc:288] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e7183525a8e942e390cc8b570a31e2fe (127.20.116.131:37981), 8f36604068874bc6a3eb9c379212b538 (127.20.116.130:44095)
Sep 28 14:56:57 dist-test-slave-pxl6 krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607416, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.131@KRBTEST.COM
Sep 28 14:56:57 dist-test-slave-pxl6 krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607416, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.130@KRBTEST.COM
I20260928 14:56:57.748826 5435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:57.738351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:37737 (local address 127.20.116.130:44095)
0928 14:56:57.738554 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:57.738567 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:57.738593 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:56:57.739082 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:57.739085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:57.739131 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:57.739256 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:57.740077 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:57.741105 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:57.742078 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:57.742484 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:57.742658 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:57.745286 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:57.745289 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:57.745290 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:57.745301 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:57.747260 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:57.747659 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:57.747661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:57.747661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:57.747788 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:57.748004 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:57.748005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:57.748006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:57.748228 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:57.748317 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:57.748584 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 14:56:57.748707 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":57,"threads_started":1}
I20260928 14:56:57.748822 5434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:57.738232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:37981 (local address 127.20.116.129:37155)
0928 14:56:57.738368 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:57.738381 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:57.738442 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:57.739349 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:57.739351 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:57.739751 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:57.739938 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:57.739944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:57.741194 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:57.741195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:57.741988 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:57.741991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:57.742640 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:57.742642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:57.742747 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:57.743395 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:57.743399 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:57.744828 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:57.747164 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:57.747172 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:57.747175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:57.747479 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:57.747758 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:57.747759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:57.747759 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:57.747886 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:57.748304 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:57.748311 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:57.748425 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 14:56:57.748527 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 14:56:57.748729 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":54,"thread_start_us":76,"threads_started":1}
I20260928 14:56:57.748903 5436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:57.738453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:37155 (local address 127.20.116.131:37981)
0928 14:56:57.738634 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 14:56:57.738644 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:56:57.738671 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:56:57.739146 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:56:57.739148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:56:57.739185 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:56:57.739343 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:56:57.740075 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:57.741098 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:57.742098 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:56:57.742484 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:56:57.742658 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:57.744945 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:56:57.744949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:56:57.744950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:56:57.744963 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:56:57.747081 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:57.747555 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:57.747559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:57.747559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:57.747672 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:56:57.747961 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:56:57.747962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:56:57.747962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:56:57.748220 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:56:57.748296 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:56:57.748600 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 14:56:57.748778 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":61,"threads_started":1}
I20260928 14:56:57.748941 5433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:57.738256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:44095 (local address 127.20.116.129:37737)
0928 14:56:57.738368 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:57.738381 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:56:57.738442 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:56:57.739346 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:56:57.739349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:56:57.739751 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:56:57.739937 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:57.739944 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:57.741220 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:57.741221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:57.741986 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:56:57.741988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:56:57.742631 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:56:57.742632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:56:57.742745 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:56:57.743328 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:56:57.743337 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:56:57.745169 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:56:57.747342 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:57.747343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:57.747343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:57.747573 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:57.747865 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:56:57.747866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:56:57.747866 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:56:57.747919 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:56:57.748337 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:56:57.748338 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:56:57.748425 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 14:56:57.748527 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 14:56:57.748812 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":143,"thread_start_us":51,"threads_started":1}
W20260928 14:56:57.749711 5048 leader_election.cc:341] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7183525a8e942e390cc8b570a31e2fe (127.20.116.131:37981): Not found: Tablet not found: b097fc4f95904ea1a6898da170a9814a
W20260928 14:56:57.750396 5046 leader_election.cc:341] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f36604068874bc6a3eb9c379212b538 (127.20.116.130:44095): Not found: Tablet not found: b097fc4f95904ea1a6898da170a9814a
I20260928 14:56:57.750481 5046 leader_election.cc:302] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cafa23b6804de7a072f74483c2719c; no voters: 8f36604068874bc6a3eb9c379212b538, e7183525a8e942e390cc8b570a31e2fe
I20260928 14:56:57.750622 5432 raft_consensus.cc:2726] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:56:58.178503 5434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:58.178346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.129:33683)
0928 14:56:58.178412 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:58.178457 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 14:56:58.223404 5437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:58.223126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.130:52417)
0928 14:56:58.223299 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:58.223353 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":90,"threads_started":1}
I20260928 14:56:58.370609 5438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:58.370347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.131:35977)
0928 14:56:58.370474 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:58.370513 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260928 14:56:59.179374 5439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:59.179068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.129:35319)
0928 14:56:59.179254 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:59.179305 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":94,"threads_started":1}
I20260928 14:56:59.224030 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:59.223820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.130:35903)
0928 14:56:59.223946 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:59.223986 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260928 14:56:59.371367 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:56:59.371122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.131:41995)
0928 14:56:59.371276 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 14:56:59.371322 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20260928 14:56:59.865720 5442 raft_consensus.cc:482] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:56:59.865872 5442 raft_consensus.cc:504] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:56:59.866082 5442 leader_election.cc:288] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e7183525a8e942e390cc8b570a31e2fe (127.20.116.131:37981), 8f36604068874bc6a3eb9c379212b538 (127.20.116.130:44095)
W20260928 14:56:59.866590 5048 leader_election.cc:341] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7183525a8e942e390cc8b570a31e2fe (127.20.116.131:37981): Not found: Tablet not found: b097fc4f95904ea1a6898da170a9814a
W20260928 14:56:59.866695 5046 leader_election.cc:341] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f36604068874bc6a3eb9c379212b538 (127.20.116.130:44095): Not found: Tablet not found: b097fc4f95904ea1a6898da170a9814a
I20260928 14:56:59.866757 5046 leader_election.cc:302] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cafa23b6804de7a072f74483c2719c; no voters: 8f36604068874bc6a3eb9c379212b538, e7183525a8e942e390cc8b570a31e2fe
I20260928 14:56:59.866840 5442 raft_consensus.cc:2726] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:57:00.180166 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:00.179911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.129:55143)
0928 14:57:00.180064 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:00.180106 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260928 14:57:00.224786 5444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:00.224509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.130:40623)
0928 14:57:00.224679 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:00.224721 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":77,"threads_started":1}
I20260928 14:57:00.372193 5445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:00.371946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.131:46793)
0928 14:57:00.372090 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:00.372140 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260928 14:57:01.180919 5446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.180698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.129:41897)
0928 14:57:01.180829 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:01.180868 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260928 14:57:01.225560 5447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.225322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.130:59105)
0928 14:57:01.225472 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:01.225513 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260928 14:57:01.372977 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:01.372731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.131:44503)
0928 14:57:01.372883 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:01.372929 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260928 14:57:02.181758 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:02.181497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.129:50557)
0928 14:57:02.181658 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:02.181706 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":92,"threads_started":1}
I20260928 14:57:02.226295 5450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:02.226045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.130:48253)
0928 14:57:02.226201 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:02.226247 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":81,"threads_started":1}
I20260928 14:57:02.373673 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:02.373472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.131:52555)
0928 14:57:02.373594 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:02.373626 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260928 14:57:03.182708 5452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.182376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.129:48197)
0928 14:57:03.182607 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:03.182654 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":87,"threads_started":1}
I20260928 14:57:03.226995 5453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.226776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.130:35399)
0928 14:57:03.226912 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:03.226953 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260928 14:57:03.232079 5454 raft_consensus.cc:482] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:57:03.232200 5454 raft_consensus.cc:504] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:57:03.232394 5454 leader_election.cc:288] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e7183525a8e942e390cc8b570a31e2fe (127.20.116.131:37981), 8f36604068874bc6a3eb9c379212b538 (127.20.116.130:44095)
W20260928 14:57:03.232784 5046 leader_election.cc:341] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f36604068874bc6a3eb9c379212b538 (127.20.116.130:44095): Not found: Tablet not found: b097fc4f95904ea1a6898da170a9814a
W20260928 14:57:03.232904 5048 leader_election.cc:341] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7183525a8e942e390cc8b570a31e2fe (127.20.116.131:37981): Not found: Tablet not found: b097fc4f95904ea1a6898da170a9814a
I20260928 14:57:03.232965 5048 leader_election.cc:302] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cafa23b6804de7a072f74483c2719c; no voters: 8f36604068874bc6a3eb9c379212b538, e7183525a8e942e390cc8b570a31e2fe
I20260928 14:57:03.233058 5454 raft_consensus.cc:2726] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:57:03.374464 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:03.374227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.131:51749)
0928 14:57:03.374365 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:03.374412 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
I20260928 14:57:04.183480 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:04.183206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.129:37689)
0928 14:57:04.183384 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:04.183427 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":81,"threads_started":1}
I20260928 14:57:04.227695 5457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:04.227453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.130:43523)
0928 14:57:04.227606 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:04.227647 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20260928 14:57:04.375260 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:04.374992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.131:52249)
0928 14:57:04.375157 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:04.375212 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":63,"threads_started":1}
I20260928 14:57:05.184363 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:05.184087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.129:39623)
0928 14:57:05.184263 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:05.184311 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":78,"threads_started":1}
I20260928 14:57:05.228533 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:05.228209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.130:54181)
0928 14:57:05.228435 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:05.228484 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":136,"threads_started":1}
I20260928 14:57:05.376016 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:05.375771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.131:49697)
0928 14:57:05.375925 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:05.375968 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":64,"threads_started":1}
I20260928 14:57:06.185129 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:06.184913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.129:37317)
0928 14:57:06.185037 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:06.185082 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260928 14:57:06.229234 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:06.228994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.130:41699)
0928 14:57:06.229144 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:06.229189 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
I20260928 14:57:06.376780 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:06.376498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.131:43423)
0928 14:57:06.376686 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:06.376729 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":76,"threads_started":1}
I20260928 14:57:07.185922 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.185692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.129:57109)
0928 14:57:07.185839 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:07.185873 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260928 14:57:07.229928 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.229688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.130:40653)
0928 14:57:07.229840 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:07.229881 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":105,"threads_started":1}
I20260928 14:57:07.377562 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:07.377328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.131:54079)
0928 14:57:07.377471 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:07.377511 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260928 14:57:07.602133 5468 raft_consensus.cc:482] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:57:07.602260 5468 raft_consensus.cc:504] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7183525a8e942e390cc8b570a31e2fe" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37981 } } peers { permanent_uuid: "42cafa23b6804de7a072f74483c2719c" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35713 } } peers { permanent_uuid: "8f36604068874bc6a3eb9c379212b538" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 44095 } }
I20260928 14:57:07.602478 5468 leader_election.cc:288] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e7183525a8e942e390cc8b570a31e2fe (127.20.116.131:37981), 8f36604068874bc6a3eb9c379212b538 (127.20.116.130:44095)
W20260928 14:57:07.602847 5048 leader_election.cc:341] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7183525a8e942e390cc8b570a31e2fe (127.20.116.131:37981): Not found: Tablet not found: b097fc4f95904ea1a6898da170a9814a
W20260928 14:57:07.602938 5046 leader_election.cc:341] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f36604068874bc6a3eb9c379212b538 (127.20.116.130:44095): Not found: Tablet not found: b097fc4f95904ea1a6898da170a9814a
I20260928 14:57:07.603026 5046 leader_election.cc:302] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cafa23b6804de7a072f74483c2719c; no voters: 8f36604068874bc6a3eb9c379212b538, e7183525a8e942e390cc8b570a31e2fe
I20260928 14:57:07.603097 5468 raft_consensus.cc:2726] T b097fc4f95904ea1a6898da170a9814a P 42cafa23b6804de7a072f74483c2719c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:57:08.186751 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:08.186457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.129:59319)
0928 14:57:08.186615 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:08.186698 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":121,"threads_started":1}
I20260928 14:57:08.230789 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:08.230495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.130:44175)
0928 14:57:08.230685 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:08.230731 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":130,"threads_started":1}
I20260928 14:57:08.378360 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:08.378106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.131:49737)
0928 14:57:08.378264 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:08.378312 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
I20260928 14:57:09.187603 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:09.187329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.129:58055)
0928 14:57:09.187503 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:09.187550 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":110,"threads_started":1}
I20260928 14:57:09.231555 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:09.231322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.130:42739)
0928 14:57:09.231467 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:09.231506 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260928 14:57:09.379215 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:09.378929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.131:44291)
0928 14:57:09.379080 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:09.379141 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260928 14:57:10.188351 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:10.188106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.129:38169)
0928 14:57:10.188259 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:10.188301 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
I20260928 14:57:10.232303 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:10.232064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.130:47083)
0928 14:57:10.232197 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:10.232253 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260928 14:57:10.379982 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:10.379739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.131:39557)
0928 14:57:10.379884 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:10.379933 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260928 14:57:11.189376 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:11.189137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.129:49921)
0928 14:57:11.189289 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:11.189329 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":66,"threads_started":1}
I20260928 14:57:11.233148 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:11.232886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.130:38689)
0928 14:57:11.233051 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:11.233101 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":101,"threads_started":1}
I20260928 14:57:11.380965 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:11.380644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40571 (local address 127.20.116.131:50519)
0928 14:57:11.380829 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:11.380889 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.190:40571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":98,"threads_started":1}
I20260928 14:57:11.381366 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 4994
I20260928 14:57:11.386512 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 5163
I20260928 14:57:11.391096 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 5296
2026-09-28T14:57:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58358 ms)
[----------] 30 tests from SecurityITest (238602 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:11 dist-test-slave-pxl6 krb5kdc[5483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:11 dist-test-slave-pxl6 krb5kdc[5483](info): set up 2 sockets
Sep 28 14:57:11 dist-test-slave-pxl6 krb5kdc[5483](info): commencing operation
krb5kdc: starting...
W20260928 14:57:12.427251 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:12 dist-test-slave-pxl6 krb5kdc[5483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:57:12.549983 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:35315
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:35315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:12.632651 5496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:12.632941 5496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:12.632979 5496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:12.634244 5496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:12.634325 5496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:12.634351 5496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:12.634373 5496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:12.634399 5496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:12.636029 5496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:35315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:35315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:12.636337 5496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:12.636605 5496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:57:12.642078 5496 server_base.cc:1034] running on GCE node
W20260928 14:57:12.642026 5502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:12.642058 5504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:12.642026 5501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:12.642486 5496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:12.642891 5496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:12.644059 5496 hybrid_clock.cc:648] HybridClock initialized: now 1790607432644027 us; error 44 us; skew 500 ppm
Sep 28 14:57:12 dist-test-slave-pxl6 krb5kdc[5483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607432, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:12.651793 5496 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:57:12.652387 5496 webserver.cc:460] Webserver started at http://127.20.116.190:36671/ using document root <none> and password file <none>
I20260928 14:57:12.652652 5496 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:12.652704 5496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:12.652802 5496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:12.653903 5496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "fe6a5af222e645479a22b569fb1a961e"
format_stamp: "Formatted at 2026-09-28 14:57:12 on dist-test-slave-pxl6"
server_key: "6a1d9420e4e31fb5c28e2e9012c63897"
server_key_iv: "c88c0c628ba68d1749155dbaab9a75a7"
server_key_version: "encryptionkey@0"
I20260928 14:57:12.654223 5496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "fe6a5af222e645479a22b569fb1a961e"
format_stamp: "Formatted at 2026-09-28 14:57:12 on dist-test-slave-pxl6"
server_key: "6a1d9420e4e31fb5c28e2e9012c63897"
server_key_iv: "c88c0c628ba68d1749155dbaab9a75a7"
server_key_version: "encryptionkey@0"
I20260928 14:57:12.655803 5496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:57:12.656629 5511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:12.656903 5496 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:57:12.656987 5496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "fe6a5af222e645479a22b569fb1a961e"
format_stamp: "Formatted at 2026-09-28 14:57:12 on dist-test-slave-pxl6"
server_key: "6a1d9420e4e31fb5c28e2e9012c63897"
server_key_iv: "c88c0c628ba68d1749155dbaab9a75a7"
server_key_version: "encryptionkey@0"
I20260928 14:57:12.657088 5496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:12.675640 5496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:12.677717 5496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:12.677874 5496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:12.682168 5496 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:35315
I20260928 14:57:12.682204 5563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:35315 every 8 connection(s)
I20260928 14:57:12.682590 5496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:12.683647 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:12.685074 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 5496
I20260928 14:57:12.685190 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:12.685400 20946 external_mini_cluster.cc:1428] Setting key 4037be0acec9359fe8a404ba38ec12bd
I20260928 14:57:12.686650 5564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e: Bootstrap starting.
I20260928 14:57:12.687399 5564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:12.687753 5564 log.cc:824] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:12.688467 5564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e: No bootstrap required, opened a new log
I20260928 14:57:12.690928 5564 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6a5af222e645479a22b569fb1a961e" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35315 } }
I20260928 14:57:12.691103 5564 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:12.691143 5564 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe6a5af222e645479a22b569fb1a961e, State: Initialized, Role: FOLLOWER
I20260928 14:57:12.691293 5564 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6a5af222e645479a22b569fb1a961e" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35315 } }
I20260928 14:57:12.691383 5564 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:12.691419 5564 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:12.691475 5564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:12.692157 5564 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6a5af222e645479a22b569fb1a961e" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35315 } }
I20260928 14:57:12.692248 5564 leader_election.cc:302] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe6a5af222e645479a22b569fb1a961e; no voters:
I20260928 14:57:12.692428 5564 leader_election.cc:288] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e [CANDIDATE]: Term 1 election: Requested vote from peers
Sep 28 14:57:12 dist-test-slave-pxl6 krb5kdc[5483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:57:12.692476 5569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:12.692662 5569 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e [term 1 LEADER]: Becoming Leader. State: Replica: fe6a5af222e645479a22b569fb1a961e, State: Running, Role: LEADER
I20260928 14:57:12.692721 5564 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:12.692792 5569 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6a5af222e645479a22b569fb1a961e" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35315 } }
I20260928 14:57:12.693286 5569 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe6a5af222e645479a22b569fb1a961e. Latest consensus state: current_term: 1 leader_uuid: "fe6a5af222e645479a22b569fb1a961e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6a5af222e645479a22b569fb1a961e" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35315 } } }
I20260928 14:57:12.693454 5570 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe6a5af222e645479a22b569fb1a961e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6a5af222e645479a22b569fb1a961e" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35315 } } }
I20260928 14:57:12.693538 5570 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:12.693442 5569 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:12.694013 5578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:12.694653 5578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:12.696239 5578 catalog_manager.cc:1347] Generated new cluster ID: fd75d7d83a14456592ae110d2de509c9
I20260928 14:57:12.696295 5578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:12.697530 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.685937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36980 (local address 127.20.116.190:35315)
0928 14:57:12.686119 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:12.686127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:12.686152 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:57:12.686630 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:12.686638 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:12.686688 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:12.686872 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:12.687622 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.688616 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.689666 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.690075 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.690156 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.693094 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:12.693098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:12.693099 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:12.693112 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:12.695539 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:12.695972 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:12.695975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:12.695975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:12.696099 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:12.696326 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:12.696327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:12.696327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:12.696642 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:12.696780 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:12.697191 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 14:57:12.697323 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260928 14:57:12.704197 5578 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:12.704944 5578 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:12.710155 5578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e: Generated new TSK 0
I20260928 14:57:12.710323 5578 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
I20260928 14:57:12.745322 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35315
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:12.829828 5590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:12.830088 5590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:12.830137 5590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:12.831389 5590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:12.831459 5590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:12.831507 5590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:57:12.833025 5590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:12.833307 5590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:12.833580 5590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:12.838901 5595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:12.838919 5596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:12.838908 5598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:12.839241 5590 server_base.cc:1034] running on GCE node
I20260928 14:57:12.839493 5590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:12.839795 5590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:12.840966 5590 hybrid_clock.cc:648] HybridClock initialized: now 1790607432840962 us; error 38 us; skew 500 ppm
Sep 28 14:57:12 dist-test-slave-pxl6 krb5kdc[5483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607432, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:12.843686 5590 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:57:12.844157 5590 webserver.cc:460] Webserver started at http://127.20.116.129:39019/ using document root <none> and password file <none>
I20260928 14:57:12.844348 5590 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:12.844388 5590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:12.844478 5590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:12.845518 5590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "c85f039f07b14bbaae21ea09e6e749b3"
format_stamp: "Formatted at 2026-09-28 14:57:12 on dist-test-slave-pxl6"
server_key: "778e4062af3277bc9926bb6fd1478cac"
server_key_iv: "732ec6ae59544f00c02e7808d4c72d3a"
server_key_version: "encryptionkey@0"
I20260928 14:57:12.845847 5590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "c85f039f07b14bbaae21ea09e6e749b3"
format_stamp: "Formatted at 2026-09-28 14:57:12 on dist-test-slave-pxl6"
server_key: "778e4062af3277bc9926bb6fd1478cac"
server_key_iv: "732ec6ae59544f00c02e7808d4c72d3a"
server_key_version: "encryptionkey@0"
I20260928 14:57:12.847295 5590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 14:57:12.848147 5605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:12.848366 5590 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:57:12.848459 5590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "c85f039f07b14bbaae21ea09e6e749b3"
format_stamp: "Formatted at 2026-09-28 14:57:12 on dist-test-slave-pxl6"
server_key: "778e4062af3277bc9926bb6fd1478cac"
server_key_iv: "732ec6ae59544f00c02e7808d4c72d3a"
server_key_version: "encryptionkey@0"
I20260928 14:57:12.848516 5590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:12.878960 5590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:12.880891 5590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:12.881006 5590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:12.881207 5590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:12.881556 5590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:12.881588 5590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:12.881635 5590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:12.881664 5590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:12.887307 5590 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:34343
I20260928 14:57:12.887362 5718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:34343 every 8 connection(s)
I20260928 14:57:12.887669 5590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:57:12.891043 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 5590
I20260928 14:57:12.891166 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:57:12.891366 20946 external_mini_cluster.cc:1428] Setting key 5da46a4885185d96b30c9145fb6da686
Sep 28 14:57:12 dist-test-slave-pxl6 krb5kdc[5483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607432, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:57:12.897878 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.888433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:41159 (local address 127.20.116.190:35315)
0928 14:57:12.888481 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:12.888482 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:12.888669 (+ 187us) server_negotiation.cc:407] Connection header received
0928 14:57:12.889153 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:12.889154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:12.889165 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:12.889217 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:12.890123 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.890697 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.891908 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.892214 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.892304 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.894880 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:12.894882 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:12.894883 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:12.894891 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:12.896415 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:12.896880 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:12.896881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:12.896882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:12.896944 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:12.897207 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:12.897208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:12.897209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:12.897336 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:12.897413 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:12.897702 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 14:57:12.897788 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:57:12.898084 5721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:12.888403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35315 (local address 127.20.116.129:41159)
0928 14:57:12.888560 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:12.888597 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:12.889066 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:12.889287 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:12.889290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:12.889850 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:12.890045 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:12.890050 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.890789 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.890790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:12.891784 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:12.891787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:12.892318 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:12.892319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:12.892449 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:12.892985 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:12.892993 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:12.894776 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:12.896521 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:12.896529 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:12.896531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:12.896799 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:12.897013 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:12.897014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:12.897015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:12.897137 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:12.897405 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:12.897412 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:12.897557 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 14:57:12.897665 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 14:57:12.897831 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260928 14:57:12.898662 5719 heartbeater.cc:347] Connected to a master server at 127.20.116.190:35315
I20260928 14:57:12.898773 5719 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:12.898998 5719 heartbeater.cc:511] Master 127.20.116.190:35315 requested a full tablet report, sending...
I20260928 14:57:12.899665 5528 ts_manager.cc:194] Registered new tserver with Master: c85f039f07b14bbaae21ea09e6e749b3 (127.20.116.129:34343)
WARNING: no policy specified for kudu/127.20.116.130@KRBTEST.COM; defaulting to no policy
I20260928 14:57:12.900583 5528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:41159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
I20260928 14:57:12.917805 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35315
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:13.004125 5724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:13.004331 5724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:13.004355 5724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:13.005471 5724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:13.005507 5724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:13.005532 5724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:57:13.007011 5724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:13.007197 5724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:13.007368 5724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:13.012653 5732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:13.012684 5724 server_base.cc:1034] running on GCE node
W20260928 14:57:13.012681 5730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:13.012653 5729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:13.013170 5724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:13.013435 5724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:13.014626 5724 hybrid_clock.cc:648] HybridClock initialized: now 1790607433014597 us; error 40 us; skew 500 ppm
Sep 28 14:57:13 dist-test-slave-pxl6 krb5kdc[5483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607433, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:13.017187 5724 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:57:13.017674 5724 webserver.cc:460] Webserver started at http://127.20.116.130:45855/ using document root <none> and password file <none>
I20260928 14:57:13.017874 5724 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:13.017915 5724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:13.018005 5724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:13.019106 5724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "3e15eb86f8d44dc59351574e85a92b96"
format_stamp: "Formatted at 2026-09-28 14:57:13 on dist-test-slave-pxl6"
server_key: "98fbc617bba7037ac027f4f63a5dbb2f"
server_key_iv: "127274c6494db184ac7f54ee866b8c96"
server_key_version: "encryptionkey@0"
I20260928 14:57:13.019419 5724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "3e15eb86f8d44dc59351574e85a92b96"
format_stamp: "Formatted at 2026-09-28 14:57:13 on dist-test-slave-pxl6"
server_key: "98fbc617bba7037ac027f4f63a5dbb2f"
server_key_iv: "127274c6494db184ac7f54ee866b8c96"
server_key_version: "encryptionkey@0"
I20260928 14:57:13.020751 5724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:57:13.021499 5739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:13.021754 5724 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:57:13.021852 5724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "3e15eb86f8d44dc59351574e85a92b96"
format_stamp: "Formatted at 2026-09-28 14:57:13 on dist-test-slave-pxl6"
server_key: "98fbc617bba7037ac027f4f63a5dbb2f"
server_key_iv: "127274c6494db184ac7f54ee866b8c96"
server_key_version: "encryptionkey@0"
I20260928 14:57:13.021907 5724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:13.047601 5724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:13.049906 5724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:13.050027 5724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:13.050246 5724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:13.050673 5724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:13.050719 5724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:13.050763 5724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:13.050791 5724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:13.057356 5724 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:34573
I20260928 14:57:13.057402 5852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:34573 every 8 connection(s)
I20260928 14:57:13.057734 5724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:57:13.063295 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 5724
I20260928 14:57:13.063416 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:57:13.063657 20946 external_mini_cluster.cc:1428] Setting key b2d1ec3d918d2950ea0ddedc10779105
Sep 28 14:57:13 dist-test-slave-pxl6 krb5kdc[5483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607433, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:57:13.068593 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.058482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:54833 (local address 127.20.116.190:35315)
0928 14:57:13.058580 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:13.058583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:13.058794 (+ 211us) server_negotiation.cc:407] Connection header received
0928 14:57:13.059350 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:13.059351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:13.059367 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:13.059422 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:13.060428 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.060992 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.062023 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.062214 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.062279 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.064976 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:13.064977 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:13.064978 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:13.064985 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:13.066909 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:13.067456 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:13.067458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:13.067458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:13.067532 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:13.067830 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:13.067831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:13.067832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:13.067967 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:13.068048 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:13.068371 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 14:57:13.068477 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:57:13.068809 5855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.058469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35315 (local address 127.20.116.130:54833)
0928 14:57:13.058662 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:13.058702 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:13.059224 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:13.059496 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:13.059500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:13.060092 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:13.060309 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.060314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.061082 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.061083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.061927 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.061929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.062299 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.062300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.062406 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.062994 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:13.063003 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:13.064871 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:13.067047 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:13.067057 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:13.067059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:13.067314 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:13.067618 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:13.067619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:13.067620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:13.067752 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:13.068050 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:13.068059 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:13.068224 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 14:57:13.068347 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 14:57:13.068532 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
I20260928 14:57:13.069676 5853 heartbeater.cc:347] Connected to a master server at 127.20.116.190:35315
I20260928 14:57:13.069830 5853 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:13.070065 5853 heartbeater.cc:511] Master 127.20.116.190:35315 requested a full tablet report, sending...
I20260928 14:57:13.070570 5528 ts_manager.cc:194] Registered new tserver with Master: 3e15eb86f8d44dc59351574e85a92b96 (127.20.116.130:34573)
I20260928 14:57:13.071130 5528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:54833
WARNING: no policy specified for kudu/127.20.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
I20260928 14:57:13.089775 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35315
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:13.172991 5858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:13.173213 5858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:13.173237 5858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:13.174441 5858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:13.174489 5858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:13.174515 5858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:57:13.176045 5858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:35315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:13.176311 5858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:13.176558 5858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:13.181563 5864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:13.181708 5858 server_base.cc:1034] running on GCE node
W20260928 14:57:13.181584 5863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:13.181802 5866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:13.182000 5858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:13.182252 5858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:13.183393 5858 hybrid_clock.cc:648] HybridClock initialized: now 1790607433183375 us; error 41 us; skew 500 ppm
Sep 28 14:57:13 dist-test-slave-pxl6 krb5kdc[5483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607433, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:13.185943 5858 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:57:13.186405 5858 webserver.cc:460] Webserver started at http://127.20.116.131:42199/ using document root <none> and password file <none>
I20260928 14:57:13.186669 5858 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:13.186723 5858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:13.186815 5858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:13.187825 5858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "740ce31be27243c7ac734e727cc5f253"
format_stamp: "Formatted at 2026-09-28 14:57:13 on dist-test-slave-pxl6"
server_key: "baffdf9b8a16a9604199cc3435651252"
server_key_iv: "96e9dd310f03e4b667603727eebd3a04"
server_key_version: "encryptionkey@0"
I20260928 14:57:13.188129 5858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "740ce31be27243c7ac734e727cc5f253"
format_stamp: "Formatted at 2026-09-28 14:57:13 on dist-test-slave-pxl6"
server_key: "baffdf9b8a16a9604199cc3435651252"
server_key_iv: "96e9dd310f03e4b667603727eebd3a04"
server_key_version: "encryptionkey@0"
I20260928 14:57:13.189499 5858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:57:13.190277 5873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:13.190482 5858 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:57:13.190605 5858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "740ce31be27243c7ac734e727cc5f253"
format_stamp: "Formatted at 2026-09-28 14:57:13 on dist-test-slave-pxl6"
server_key: "baffdf9b8a16a9604199cc3435651252"
server_key_iv: "96e9dd310f03e4b667603727eebd3a04"
server_key_version: "encryptionkey@0"
I20260928 14:57:13.190685 5858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:13.215380 5858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:13.217381 5858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:13.217541 5858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:13.217787 5858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:13.218140 5858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:13.218195 5858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:13.218231 5858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:13.218261 5858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:13.224267 5858 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:34827
I20260928 14:57:13.224309 5986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:34827 every 8 connection(s)
I20260928 14:57:13.224699 5858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:57:13.224982 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 5858
I20260928 14:57:13.225078 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:57:13.225263 20946 external_mini_cluster.cc:1428] Setting key 90d5f5b1a03c834a6bb3e61e1f4f3878
Sep 28 14:57:13 dist-test-slave-pxl6 krb5kdc[5483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607433, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:57:13.234351 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.225468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:53835 (local address 127.20.116.190:35315)
0928 14:57:13.225526 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:13.225528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:13.225857 (+ 329us) server_negotiation.cc:407] Connection header received
0928 14:57:13.226505 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:13.226506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:13.226518 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:13.226587 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:13.227590 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.228108 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.229161 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.229374 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.229460 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.231770 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:13.231772 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:13.231772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:13.231779 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:13.233207 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:13.233547 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:13.233548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:13.233548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:13.233599 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:13.233836 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:13.233836 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:13.233837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:13.233939 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:13.233990 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:13.234219 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 14:57:13.234280 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:57:13.234509 5989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.225458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:35315 (local address 127.20.116.131:53835)
0928 14:57:13.225689 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:13.225757 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:13.226400 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:13.226663 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:13.226667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:13.227260 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:13.227475 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.227481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.228209 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.228210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.229054 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.229056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.229472 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.229473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.229609 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.230179 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:13.230187 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:13.231680 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:13.233276 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:13.233282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:13.233284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:13.233484 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:13.233657 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:13.233657 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:13.233658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:13.233760 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:13.233993 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:13.233998 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:13.234110 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 14:57:13.234191 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 14:57:13.234306 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":102,"threads_started":1}
I20260928 14:57:13.235011 5987 heartbeater.cc:347] Connected to a master server at 127.20.116.190:35315
I20260928 14:57:13.235098 5987 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:13.235255 5987 heartbeater.cc:511] Master 127.20.116.190:35315 requested a full tablet report, sending...
I20260928 14:57:13.235607 5528 ts_manager.cc:194] Registered new tserver with Master: 740ce31be27243c7ac734e727cc5f253 (127.20.116.131:34827)
I20260928 14:57:13.235986 5528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:53835
I20260928 14:57:13.236979 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:57:13.244454 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.238278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36982 (local address 127.20.116.190:35315)
0928 14:57:13.238317 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:13.238318 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:13.238475 (+ 157us) server_negotiation.cc:407] Connection header received
0928 14:57:13.238567 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:13.238568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:13.238578 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:13.238649 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:13.239201 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.239662 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.240409 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.240596 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.240668 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.241846 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:13.241848 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:13.241848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:13.241856 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:13.243272 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:13.243639 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:13.243640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:13.243641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:13.243702 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:13.243906 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:13.243907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:13.243908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:13.244061 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:13.244190 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:13.244274 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 14:57:13.244350 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:57:13.246184 5528 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:57:13.247823 5528 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:57:13.254817 6000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.250206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:34573 (local address 127.0.0.1:46916)
0928 14:57:13.250601 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:13.250646 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:13.250719 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:13.250993 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:13.250996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:13.250999 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:13.251074 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.251076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.252445 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.252446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.253713 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.253716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.254455 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.254457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.254565 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.254584 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 14:57:13.254660 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 14:57:13.254727 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":56,"thread_start_us":99,"threads_started":1}
I20260928 14:57:13.254841 6003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.250467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46916 (local address 127.20.116.130:34573)
0928 14:57:13.250685 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:13.250690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:13.250701 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:57:13.250861 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:13.250864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:13.250882 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:13.250990 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:13.251168 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.252341 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.253820 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.254362 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.254459 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.254521 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:13.254689 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 14:57:13.254750 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":105,"threads_started":1}
I20260928 14:57:13.255062 6002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.250407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58932 (local address 127.20.116.129:34343)
0928 14:57:13.250679 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:13.250683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:13.250694 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:57:13.250847 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:13.250849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:13.250933 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:13.251165 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:13.251279 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.252378 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.253695 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.254412 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.254488 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.254561 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:13.254867 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 14:57:13.254976 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":78,"threads_started":1}
I20260928 14:57:13.255077 5999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.250205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:34827 (local address 127.0.0.1:42338)
0928 14:57:13.250868 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:13.250877 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:13.250958 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:13.251218 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:13.251219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:13.251220 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:13.251261 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.251262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.252248 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.252249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.253843 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.253845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.254810 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.254811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.254878 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.254883 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:57:13.254965 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 14:57:13.255010 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":131,"thread_start_us":93,"threads_started":1}
I20260928 14:57:13.255288 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.250371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42338 (local address 127.20.116.131:34827)
0928 14:57:13.250591 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:13.250597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:13.250930 (+ 333us) server_negotiation.cc:407] Connection header received
0928 14:57:13.251053 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:13.251056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:13.251071 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:13.251122 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:13.251324 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.252149 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.253925 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.254656 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.254726 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.254774 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:13.254989 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 14:57:13.255190 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
I20260928 14:57:13.254824 5998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.250207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:34343 (local address 127.0.0.1:58932)
0928 14:57:13.250603 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:13.250646 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:13.250729 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:13.251158 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:13.251159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:13.251161 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:13.251215 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.251217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.252482 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.252483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.253606 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.253609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.254591 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.254592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.254663 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.254666 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:57:13.254714 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 14:57:13.254744 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":44,"threads_started":1}
I20260928 14:57:13.255961 5787 tablet_service.cc:1467] Processing CreateTablet for tablet 6a71ef8c67f34df69354126179931515 (DEFAULT_TABLE table=test-table [id=4cc9b2606c08406ca82ae8510d2d524f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:13.256361 5787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a71ef8c67f34df69354126179931515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:13.256314 5653 tablet_service.cc:1467] Processing CreateTablet for tablet 6a71ef8c67f34df69354126179931515 (DEFAULT_TABLE table=test-table [id=4cc9b2606c08406ca82ae8510d2d524f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:13.256661 5653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a71ef8c67f34df69354126179931515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:13.259267 5921 tablet_service.cc:1467] Processing CreateTablet for tablet 6a71ef8c67f34df69354126179931515 (DEFAULT_TABLE table=test-table [id=4cc9b2606c08406ca82ae8510d2d524f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:13.259606 5921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a71ef8c67f34df69354126179931515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:13.260103 6004 tablet_bootstrap.cc:492] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3: Bootstrap starting.
I20260928 14:57:13.260820 6004 tablet_bootstrap.cc:654] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:13.261435 6004 log.cc:824] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:13.261804 6005 tablet_bootstrap.cc:492] T 6a71ef8c67f34df69354126179931515 P 3e15eb86f8d44dc59351574e85a92b96: Bootstrap starting.
I20260928 14:57:13.262187 6004 tablet_bootstrap.cc:492] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3: No bootstrap required, opened a new log
I20260928 14:57:13.262272 6004 ts_tablet_manager.cc:1397] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:57:13.262495 6005 tablet_bootstrap.cc:654] T 6a71ef8c67f34df69354126179931515 P 3e15eb86f8d44dc59351574e85a92b96: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:13.263118 6005 log.cc:824] T 6a71ef8c67f34df69354126179931515 P 3e15eb86f8d44dc59351574e85a92b96: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:13.263931 6005 tablet_bootstrap.cc:492] T 6a71ef8c67f34df69354126179931515 P 3e15eb86f8d44dc59351574e85a92b96: No bootstrap required, opened a new log
I20260928 14:57:13.263985 6005 ts_tablet_manager.cc:1397] T 6a71ef8c67f34df69354126179931515 P 3e15eb86f8d44dc59351574e85a92b96: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:57:13.264267 6004 raft_consensus.cc:348] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "740ce31be27243c7ac734e727cc5f253" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34827 } } peers { permanent_uuid: "c85f039f07b14bbaae21ea09e6e749b3" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34343 } } peers { permanent_uuid: "3e15eb86f8d44dc59351574e85a92b96" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 34573 } }
I20260928 14:57:13.264436 6004 raft_consensus.cc:374] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:13.264482 6004 raft_consensus.cc:729] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c85f039f07b14bbaae21ea09e6e749b3, State: Initialized, Role: FOLLOWER
I20260928 14:57:13.264602 6004 consensus_queue.cc:260] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "740ce31be27243c7ac734e727cc5f253" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34827 } } peers { permanent_uuid: "c85f039f07b14bbaae21ea09e6e749b3" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34343 } } peers { permanent_uuid: "3e15eb86f8d44dc59351574e85a92b96" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 34573 } }
I20260928 14:57:13.264914 6004 ts_tablet_manager.cc:1428] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:57:13.264940 5719 heartbeater.cc:503] Master 127.20.116.190:35315 was elected leader, sending a full tablet report...
I20260928 14:57:13.265849 6005 raft_consensus.cc:348] T 6a71ef8c67f34df69354126179931515 P 3e15eb86f8d44dc59351574e85a92b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "740ce31be27243c7ac734e727cc5f253" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34827 } } peers { permanent_uuid: "c85f039f07b14bbaae21ea09e6e749b3" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34343 } } peers { permanent_uuid: "3e15eb86f8d44dc59351574e85a92b96" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 34573 } }
I20260928 14:57:13.266048 6005 raft_consensus.cc:374] T 6a71ef8c67f34df69354126179931515 P 3e15eb86f8d44dc59351574e85a92b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:13.266100 6005 raft_consensus.cc:729] T 6a71ef8c67f34df69354126179931515 P 3e15eb86f8d44dc59351574e85a92b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e15eb86f8d44dc59351574e85a92b96, State: Initialized, Role: FOLLOWER
I20260928 14:57:13.266212 6005 consensus_queue.cc:260] T 6a71ef8c67f34df69354126179931515 P 3e15eb86f8d44dc59351574e85a92b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "740ce31be27243c7ac734e727cc5f253" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34827 } } peers { permanent_uuid: "c85f039f07b14bbaae21ea09e6e749b3" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34343 } } peers { permanent_uuid: "3e15eb86f8d44dc59351574e85a92b96" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 34573 } }
I20260928 14:57:13.266510 5853 heartbeater.cc:503] Master 127.20.116.190:35315 was elected leader, sending a full tablet report...
I20260928 14:57:13.266511 6005 ts_tablet_manager.cc:1428] T 6a71ef8c67f34df69354126179931515 P 3e15eb86f8d44dc59351574e85a92b96: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 14:57:13.266677 6008 tablet_bootstrap.cc:492] T 6a71ef8c67f34df69354126179931515 P 740ce31be27243c7ac734e727cc5f253: Bootstrap starting.
I20260928 14:57:13.267356 6008 tablet_bootstrap.cc:654] T 6a71ef8c67f34df69354126179931515 P 740ce31be27243c7ac734e727cc5f253: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:13.267930 6008 log.cc:824] T 6a71ef8c67f34df69354126179931515 P 740ce31be27243c7ac734e727cc5f253: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:13.268574 6008 tablet_bootstrap.cc:492] T 6a71ef8c67f34df69354126179931515 P 740ce31be27243c7ac734e727cc5f253: No bootstrap required, opened a new log
I20260928 14:57:13.268630 6008 ts_tablet_manager.cc:1397] T 6a71ef8c67f34df69354126179931515 P 740ce31be27243c7ac734e727cc5f253: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:57:13.269975 6008 raft_consensus.cc:348] T 6a71ef8c67f34df69354126179931515 P 740ce31be27243c7ac734e727cc5f253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "740ce31be27243c7ac734e727cc5f253" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34827 } } peers { permanent_uuid: "c85f039f07b14bbaae21ea09e6e749b3" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34343 } } peers { permanent_uuid: "3e15eb86f8d44dc59351574e85a92b96" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 34573 } }
I20260928 14:57:13.270116 6008 raft_consensus.cc:374] T 6a71ef8c67f34df69354126179931515 P 740ce31be27243c7ac734e727cc5f253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:13.270140 6008 raft_consensus.cc:729] T 6a71ef8c67f34df69354126179931515 P 740ce31be27243c7ac734e727cc5f253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 740ce31be27243c7ac734e727cc5f253, State: Initialized, Role: FOLLOWER
I20260928 14:57:13.270263 6008 consensus_queue.cc:260] T 6a71ef8c67f34df69354126179931515 P 740ce31be27243c7ac734e727cc5f253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "740ce31be27243c7ac734e727cc5f253" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34827 } } peers { permanent_uuid: "c85f039f07b14bbaae21ea09e6e749b3" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34343 } } peers { permanent_uuid: "3e15eb86f8d44dc59351574e85a92b96" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 34573 } }
I20260928 14:57:13.270512 6008 ts_tablet_manager.cc:1428] T 6a71ef8c67f34df69354126179931515 P 740ce31be27243c7ac734e727cc5f253: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:57:13.270603 5987 heartbeater.cc:503] Master 127.20.116.190:35315 was elected leader, sending a full tablet report...
W20260928 14:57:13.308332 5854 tablet.cc:2369] T 6a71ef8c67f34df69354126179931515 P 3e15eb86f8d44dc59351574e85a92b96: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:57:13.388497 5720 tablet.cc:2369] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:57:13.424261 6009 raft_consensus.cc:482] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:57:13.424435 6009 raft_consensus.cc:504] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "740ce31be27243c7ac734e727cc5f253" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34827 } } peers { permanent_uuid: "c85f039f07b14bbaae21ea09e6e749b3" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34343 } } peers { permanent_uuid: "3e15eb86f8d44dc59351574e85a92b96" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 34573 } }
I20260928 14:57:13.424880 6009 leader_election.cc:288] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 740ce31be27243c7ac734e727cc5f253 (127.20.116.131:34827), 3e15eb86f8d44dc59351574e85a92b96 (127.20.116.130:34573)
I20260928 14:57:13.427932 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.425132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:50333 (local address 127.20.116.131:34827)
0928 14:57:13.425182 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:13.425184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:13.425311 (+ 127us) server_negotiation.cc:407] Connection header received
0928 14:57:13.425381 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:13.425382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:13.425395 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:13.425466 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:13.425647 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.426133 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.427188 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.427564 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.427661 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.427671 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:13.427822 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 14:57:13.427857 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:57:13.427937 6003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.425136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:33325 (local address 127.20.116.130:34573)
0928 14:57:13.425185 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:13.425188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:13.425296 (+ 108us) server_negotiation.cc:407] Connection header received
0928 14:57:13.425394 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:13.425395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:13.425417 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:13.425476 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:13.425711 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.426182 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.427200 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.427576 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.427664 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.427673 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:13.427826 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 14:57:13.427862 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:57:13.427963 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.425073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:34827 (local address 127.20.116.129:50333)
0928 14:57:13.425220 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:13.425229 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:13.425318 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:13.425479 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:13.425479 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:13.425481 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:13.425568 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.425575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.426222 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.426223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.427095 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.427097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.427661 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.427662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.427742 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.427746 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:57:13.427815 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 14:57:13.427857 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":3,"thread_start_us":66,"threads_started":1}
I20260928 14:57:13.427963 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.425073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:34573 (local address 127.20.116.129:33325)
0928 14:57:13.425213 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:13.425224 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:13.425320 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:13.425496 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:13.425497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:13.425499 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:13.425589 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.425591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.426271 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.426273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.427103 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.427106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.427664 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.427665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.427744 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.427748 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:57:13.427817 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 14:57:13.427860 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260928 14:57:13.428308 5941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a71ef8c67f34df69354126179931515" candidate_uuid: "c85f039f07b14bbaae21ea09e6e749b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "740ce31be27243c7ac734e727cc5f253" is_pre_election: true
I20260928 14:57:13.428438 5941 raft_consensus.cc:2454] T 6a71ef8c67f34df69354126179931515 P 740ce31be27243c7ac734e727cc5f253 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c85f039f07b14bbaae21ea09e6e749b3 in term 0.
I20260928 14:57:13.428675 5607 leader_election.cc:302] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 740ce31be27243c7ac734e727cc5f253, c85f039f07b14bbaae21ea09e6e749b3; no voters:
I20260928 14:57:13.428711 5807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a71ef8c67f34df69354126179931515" candidate_uuid: "c85f039f07b14bbaae21ea09e6e749b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e15eb86f8d44dc59351574e85a92b96" is_pre_election: true
I20260928 14:57:13.428856 5807 raft_consensus.cc:2454] T 6a71ef8c67f34df69354126179931515 P 3e15eb86f8d44dc59351574e85a92b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c85f039f07b14bbaae21ea09e6e749b3 in term 0.
I20260928 14:57:13.428908 6009 raft_consensus.cc:2781] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:57:13.428975 6009 raft_consensus.cc:482] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:57:13.429015 6009 raft_consensus.cc:3037] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:13.429701 6009 raft_consensus.cc:504] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "740ce31be27243c7ac734e727cc5f253" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34827 } } peers { permanent_uuid: "c85f039f07b14bbaae21ea09e6e749b3" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34343 } } peers { permanent_uuid: "3e15eb86f8d44dc59351574e85a92b96" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 34573 } }
I20260928 14:57:13.429862 6009 leader_election.cc:288] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3 [CANDIDATE]: Term 1 election: Requested vote from peers 740ce31be27243c7ac734e727cc5f253 (127.20.116.131:34827), 3e15eb86f8d44dc59351574e85a92b96 (127.20.116.130:34573)
I20260928 14:57:13.430019 5941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a71ef8c67f34df69354126179931515" candidate_uuid: "c85f039f07b14bbaae21ea09e6e749b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "740ce31be27243c7ac734e727cc5f253"
I20260928 14:57:13.430073 5807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a71ef8c67f34df69354126179931515" candidate_uuid: "c85f039f07b14bbaae21ea09e6e749b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e15eb86f8d44dc59351574e85a92b96"
I20260928 14:57:13.430127 5941 raft_consensus.cc:3037] T 6a71ef8c67f34df69354126179931515 P 740ce31be27243c7ac734e727cc5f253 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:13.430132 5807 raft_consensus.cc:3037] T 6a71ef8c67f34df69354126179931515 P 3e15eb86f8d44dc59351574e85a92b96 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:13.430949 5807 raft_consensus.cc:2454] T 6a71ef8c67f34df69354126179931515 P 3e15eb86f8d44dc59351574e85a92b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c85f039f07b14bbaae21ea09e6e749b3 in term 1.
I20260928 14:57:13.430975 5941 raft_consensus.cc:2454] T 6a71ef8c67f34df69354126179931515 P 740ce31be27243c7ac734e727cc5f253 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c85f039f07b14bbaae21ea09e6e749b3 in term 1.
I20260928 14:57:13.431139 5607 leader_election.cc:302] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 740ce31be27243c7ac734e727cc5f253, c85f039f07b14bbaae21ea09e6e749b3; no voters:
I20260928 14:57:13.431247 6009 raft_consensus.cc:2781] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:13.431417 6009 raft_consensus.cc:686] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3 [term 1 LEADER]: Becoming Leader. State: Replica: c85f039f07b14bbaae21ea09e6e749b3, State: Running, Role: LEADER
I20260928 14:57:13.431525 6009 consensus_queue.cc:237] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "740ce31be27243c7ac734e727cc5f253" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34827 } } peers { permanent_uuid: "c85f039f07b14bbaae21ea09e6e749b3" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34343 } } peers { permanent_uuid: "3e15eb86f8d44dc59351574e85a92b96" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 34573 } }
I20260928 14:57:13.432379 5527 catalog_manager.cc:5495] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c85f039f07b14bbaae21ea09e6e749b3 (127.20.116.129). New cstate: current_term: 1 leader_uuid: "c85f039f07b14bbaae21ea09e6e749b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "740ce31be27243c7ac734e727cc5f253" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c85f039f07b14bbaae21ea09e6e749b3" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e15eb86f8d44dc59351574e85a92b96" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 34573 } health_report { overall_health: UNKNOWN } } }
W20260928 14:57:13.475330 5988 tablet.cc:2369] T 6a71ef8c67f34df69354126179931515 P 740ce31be27243c7ac734e727cc5f253: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:57:13.481989 6002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.479474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58948 (local address 127.20.116.129:34343)
0928 14:57:13.479543 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:13.479545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:13.479554 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:57:13.479604 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:13.479605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:13.479627 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:13.479706 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:13.479867 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.480354 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.481155 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.481360 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.481440 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.481625 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:13.481734 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:13.481833 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:13.481873 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 14:57:13.481895 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:57:13.484164 5941 raft_consensus.cc:1261] T 6a71ef8c67f34df69354126179931515 P 740ce31be27243c7ac734e727cc5f253 [term 1 FOLLOWER]: Refusing update from remote peer c85f039f07b14bbaae21ea09e6e749b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:13.484163 5807 raft_consensus.cc:1261] T 6a71ef8c67f34df69354126179931515 P 3e15eb86f8d44dc59351574e85a92b96 [term 1 FOLLOWER]: Refusing update from remote peer c85f039f07b14bbaae21ea09e6e749b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:13.484586 6009 consensus_queue.cc:1035] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "740ce31be27243c7ac734e727cc5f253" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 34827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:13.484809 6015 consensus_queue.cc:1035] T 6a71ef8c67f34df69354126179931515 P c85f039f07b14bbaae21ea09e6e749b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e15eb86f8d44dc59351574e85a92b96" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 34573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:13.487368 6018 mvcc.cc:205] Tried to move back new op lower bound from 7334328047547691008 to 7334328047336177664. Current Snapshot: MvccSnapshot[applied={T|T < 7334328047547691008}]
I20260928 14:57:13.488427 6020 mvcc.cc:205] Tried to move back new op lower bound from 7334328047547691008 to 7334328047336177664. Current Snapshot: MvccSnapshot[applied={T|T < 7334328047547691008 or (T in {7334328047547691008})}]
I20260928 14:57:13.489559 6019 mvcc.cc:205] Tried to move back new op lower bound from 7334328047547691008 to 7334328047336177664. Current Snapshot: MvccSnapshot[applied={T|T < 7334328047547691008 or (T in {7334328047547691008})}]
I20260928 14:57:13.489701 5527 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:13.489794 5527 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:13.490561 5527 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe6a5af222e645479a22b569fb1a961e: Sending DeleteTablet for 3 replicas of tablet 6a71ef8c67f34df69354126179931515
I20260928 14:57:13.490928 5653 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a71ef8c67f34df69354126179931515 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:13 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:58932
I20260928 14:57:13.490928 5787 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a71ef8c67f34df69354126179931515 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:13 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:46916
I20260928 14:57:13.491178 6028 tablet_replica.cc:330] T 6a71ef8c67f34df69354126179931515 P 3e15eb86f8d44dc59351574e85a92b96: stopping tablet replica
I20260928 14:57:13.491158 5921 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a71ef8c67f34df69354126179931515 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:13 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:42338
I20260928 14:57:13.491315 6028 raft_consensus.cc:2229] T 6a71ef8c67f34df69354126179931515 P 3e15eb86f8d44dc59351574e85a92b96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:13.491384 6030 tablet_replica.cc:330] T 6a71ef8c67f34df69354126179931515 P 740ce31be27243c7ac734e727cc5f253: stopping tablet replica
I20260928 14:57:13.491446 6028 raft_consensus.cc:2258] T 6a71ef8c67f34df69354126179931515 P 3e15eb86f8d44dc59351574e85a92b96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:13.491508 6030 raft_consensus.cc:2229] T 6a71ef8c67f34df69354126179931515 P 740ce31be27243c7ac734e727cc5f253 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:13.491715 6030 raft_consensus.cc:2258] T 6a71ef8c67f34df69354126179931515 P 740ce31be27243c7ac734e727cc5f253 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:13.491930 6028 ts_tablet_manager.cc:1905] T 6a71ef8c67f34df69354126179931515 P 3e15eb86f8d44dc59351574e85a92b96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:13.492041 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 5590
I20260928 14:57:13.492959 6030 ts_tablet_manager.cc:1905] T 6a71ef8c67f34df69354126179931515 P 740ce31be27243c7ac734e727cc5f253: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:13.494109 6028 ts_tablet_manager.cc:1918] T 6a71ef8c67f34df69354126179931515 P 3e15eb86f8d44dc59351574e85a92b96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:13.494189 6028 log.cc:1196] T 6a71ef8c67f34df69354126179931515 P 3e15eb86f8d44dc59351574e85a92b96: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/6a71ef8c67f34df69354126179931515
I20260928 14:57:13.494468 6028 ts_tablet_manager.cc:1939] T 6a71ef8c67f34df69354126179931515 P 3e15eb86f8d44dc59351574e85a92b96: Deleting consensus metadata
I20260928 14:57:13.494622 6030 ts_tablet_manager.cc:1918] T 6a71ef8c67f34df69354126179931515 P 740ce31be27243c7ac734e727cc5f253: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:13.494731 6030 log.cc:1196] T 6a71ef8c67f34df69354126179931515 P 740ce31be27243c7ac734e727cc5f253: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/6a71ef8c67f34df69354126179931515
I20260928 14:57:13.495003 5512 catalog_manager.cc:4841] TS 3e15eb86f8d44dc59351574e85a92b96 (127.20.116.130:34573): tablet 6a71ef8c67f34df69354126179931515 (table test-table [id=4cc9b2606c08406ca82ae8510d2d524f]) successfully deleted
I20260928 14:57:13.495039 6030 ts_tablet_manager.cc:1939] T 6a71ef8c67f34df69354126179931515 P 740ce31be27243c7ac734e727cc5f253: Deleting consensus metadata
I20260928 14:57:13.495471 5513 catalog_manager.cc:4841] TS 740ce31be27243c7ac734e727cc5f253 (127.20.116.131:34827): tablet 6a71ef8c67f34df69354126179931515 (table test-table [id=4cc9b2606c08406ca82ae8510d2d524f]) successfully deleted
W20260928 14:57:13.499603 5514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.116.129:34343 (error 108)
I20260928 14:57:13.499828 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 5724
I20260928 14:57:13.501761 5998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.501552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:34343 (local address 127.0.0.1:58964)
0928 14:57:13.501620 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:13.501666 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.129:34343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 14:57:13.501914 5514 catalog_manager.cc:4584] TS c85f039f07b14bbaae21ea09e6e749b3 (127.20.116.129:34343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a71ef8c67f34df69354126179931515: Network error: Client connection negotiation failed: client connection to 127.20.116.129:34343: connect: Connection refused (error 111)
I20260928 14:57:13.505648 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 5858
I20260928 14:57:13.510332 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 5496
2026-09-28T14:57:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2119 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T14:57:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:13Z Disabled control of system clock
I20260928 14:57:13.538693 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:34017
--webserver_interface=127.20.116.190
--webserver_port=0
--builtin_ntp_servers=127.20.116.148:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:34017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:57:13.619721 6035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:13.619925 6035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:13.619946 6035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:13.621017 6035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:13.621052 6035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:13.621065 6035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:13.621078 6035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:13.621089 6035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:13.622352 6035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:34017
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:34017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:13.622623 6035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:13.622874 6035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:13.627751 6043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:13.627781 6040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:13.627740 6041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:13.628062 6035 server_base.cc:1034] running on GCE node
I20260928 14:57:13.628362 6035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:13.628669 6035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:13.629859 6035 hybrid_clock.cc:648] HybridClock initialized: now 1790607433629844 us; error 32 us; skew 500 ppm
I20260928 14:57:13.631358 6035 webserver.cc:460] Webserver started at http://127.20.116.190:40333/ using document root <none> and password file <none>
I20260928 14:57:13.631618 6035 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:13.631690 6035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:13.631827 6035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:13.632961 6035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "54aa532a740d45ff9d91532325ed6730"
format_stamp: "Formatted at 2026-09-28 14:57:13 on dist-test-slave-pxl6"
server_key: "84dbdebf9e2da7e538d73319e2d0d8c8"
server_key_iv: "2cc6ba560c0abacd0f13733db617fac0"
server_key_version: "encryptionkey@0"
I20260928 14:57:13.633337 6035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "54aa532a740d45ff9d91532325ed6730"
format_stamp: "Formatted at 2026-09-28 14:57:13 on dist-test-slave-pxl6"
server_key: "84dbdebf9e2da7e538d73319e2d0d8c8"
server_key_iv: "2cc6ba560c0abacd0f13733db617fac0"
server_key_version: "encryptionkey@0"
I20260928 14:57:13.634655 6035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:13.635433 6049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:13.635602 6035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:57:13.635706 6035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "54aa532a740d45ff9d91532325ed6730"
format_stamp: "Formatted at 2026-09-28 14:57:13 on dist-test-slave-pxl6"
server_key: "84dbdebf9e2da7e538d73319e2d0d8c8"
server_key_iv: "2cc6ba560c0abacd0f13733db617fac0"
server_key_version: "encryptionkey@0"
I20260928 14:57:13.635793 6035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:13.653995 6035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:13.654369 6035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:13.654600 6035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:13.658502 6035 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:34017
I20260928 14:57:13.658573 6101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:34017 every 8 connection(s)
I20260928 14:57:13.658936 6035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:13.659952 6102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:13.662263 6102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730: Bootstrap starting.
I20260928 14:57:13.662914 6102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:13.662989 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 6035
I20260928 14:57:13.663125 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:13.663304 6102 log.cc:824] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:13.663340 20946 external_mini_cluster.cc:1428] Setting key aef1f495b4078dcf12fd1933c8faf2e2
I20260928 14:57:13.664168 6102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730: No bootstrap required, opened a new log
I20260928 14:57:13.666968 6102 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54aa532a740d45ff9d91532325ed6730" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34017 } }
I20260928 14:57:13.667109 6102 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:13.667131 6102 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54aa532a740d45ff9d91532325ed6730, State: Initialized, Role: FOLLOWER
I20260928 14:57:13.667322 6102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54aa532a740d45ff9d91532325ed6730" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34017 } }
I20260928 14:57:13.667408 6102 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:13.667474 6102 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:13.667529 6102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:13.668283 6102 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54aa532a740d45ff9d91532325ed6730" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34017 } }
I20260928 14:57:13.668401 6102 leader_election.cc:302] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54aa532a740d45ff9d91532325ed6730; no voters:
I20260928 14:57:13.668576 6102 leader_election.cc:288] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:13.668627 6107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:13.668812 6107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730 [term 1 LEADER]: Becoming Leader. State: Replica: 54aa532a740d45ff9d91532325ed6730, State: Running, Role: LEADER
I20260928 14:57:13.668967 6107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54aa532a740d45ff9d91532325ed6730" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34017 } }
I20260928 14:57:13.669262 6102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:13.669448 6108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54aa532a740d45ff9d91532325ed6730" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54aa532a740d45ff9d91532325ed6730" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34017 } } }
I20260928 14:57:13.669695 6108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:13.669622 6109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54aa532a740d45ff9d91532325ed6730. Latest consensus state: current_term: 1 leader_uuid: "54aa532a740d45ff9d91532325ed6730" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54aa532a740d45ff9d91532325ed6730" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 34017 } } }
I20260928 14:57:13.670120 6109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:13.670245 6113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:13.670414 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.663850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60488 (local address 127.20.116.190:34017)
0928 14:57:13.664063 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:13.664073 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:13.664104 (+ 31us) server_negotiation.cc:407] Connection header received
0928 14:57:13.664730 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:13.664740 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:13.664854 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:13.665020 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:13.665190 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.666704 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.667764 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.668419 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.668557 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.669314 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:13.669319 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:13.669320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:13.669556 (+ 236us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:13.669600 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:13.669615 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:13.669697 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:13.670031 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 14:57:13.670134 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20260928 14:57:13.670794 6113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:13.672361 6113 catalog_manager.cc:1347] Generated new cluster ID: 831ae48279064116a403c1fbc24a7efc
I20260928 14:57:13.672415 6113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:13.689273 6113 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:13.689844 6113 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:13.698946 6113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730: Generated new TSK 0
I20260928 14:57:13.699159 6113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:13.705225 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34017
--builtin_ntp_servers=127.20.116.148:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:57:13.806957 6126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:13.807152 6126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:13.807173 6126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:13.808297 6126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:13.808336 6126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:13.808362 6126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:57:13.809687 6126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:13.809949 6126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:13.810261 6126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:13.815305 6134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:13.815546 6126 server_base.cc:1034] running on GCE node
W20260928 14:57:13.815308 6131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:13.815310 6132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:13.815886 6126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:13.816150 6126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:13.817317 6126 hybrid_clock.cc:648] HybridClock initialized: now 1790607433817306 us; error 34 us; skew 500 ppm
I20260928 14:57:13.818677 6126 webserver.cc:460] Webserver started at http://127.20.116.129:39271/ using document root <none> and password file <none>
I20260928 14:57:13.819005 6126 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:13.819080 6126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:13.819258 6126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:13.820344 6126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "fa3bec0cceed4cfa882167476df657da"
format_stamp: "Formatted at 2026-09-28 14:57:13 on dist-test-slave-pxl6"
server_key: "e1b0a84ec8945c8d55a68435eec97f90"
server_key_iv: "15a78f5a020eebeea7b53d5f4db5d3a2"
server_key_version: "encryptionkey@0"
I20260928 14:57:13.820801 6126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "fa3bec0cceed4cfa882167476df657da"
format_stamp: "Formatted at 2026-09-28 14:57:13 on dist-test-slave-pxl6"
server_key: "e1b0a84ec8945c8d55a68435eec97f90"
server_key_iv: "15a78f5a020eebeea7b53d5f4db5d3a2"
server_key_version: "encryptionkey@0"
I20260928 14:57:13.822184 6126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:57:13.823061 6140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:13.823238 6126 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:57:13.823340 6126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "fa3bec0cceed4cfa882167476df657da"
format_stamp: "Formatted at 2026-09-28 14:57:13 on dist-test-slave-pxl6"
server_key: "e1b0a84ec8945c8d55a68435eec97f90"
server_key_iv: "15a78f5a020eebeea7b53d5f4db5d3a2"
server_key_version: "encryptionkey@0"
I20260928 14:57:13.823426 6126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:13.839690 6126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:13.840010 6126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:13.840160 6126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:13.840395 6126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:13.840770 6126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:13.840834 6126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:13.840878 6126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:13.840905 6126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:13.846709 6126 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:35335
I20260928 14:57:13.846755 6253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:35335 every 8 connection(s)
I20260928 14:57:13.847133 6126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:57:13.850956 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 6126
I20260928 14:57:13.851046 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:57:13.851261 20946 external_mini_cluster.cc:1428] Setting key cb9a8264e2be76a77f8cae1fc4e355ba
I20260928 14:57:13.852347 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34017
--builtin_ntp_servers=127.20.116.148:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:57:13.852739 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.847881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:47629 (local address 127.20.116.190:34017)
0928 14:57:13.847956 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:13.847958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:13.848197 (+ 239us) server_negotiation.cc:407] Connection header received
0928 14:57:13.848740 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:13.848741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:13.848754 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:13.848808 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:13.849515 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.850027 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.851109 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.851334 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.851424 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.852370 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:13.852371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:13.852371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:13.852377 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:13.852392 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:13.852397 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:13.852450 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:13.852587 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 14:57:13.852615 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:57:13.852866 6256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.847883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34017 (local address 127.20.116.129:47629)
0928 14:57:13.848072 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:13.848106 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:13.848625 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:13.848903 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:13.848907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:13.848915 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:13.849415 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.849419 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.850134 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.850135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.850960 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.850962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.851433 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.851435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.851594 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.852219 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:57:13.852219 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:13.852248 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:57:13.852270 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:13.852479 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:13.852487 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:13.852491 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:57:13.852579 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 14:57:13.852664 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":71,"threads_started":1}
I20260928 14:57:13.854146 6254 heartbeater.cc:347] Connected to a master server at 127.20.116.190:34017
I20260928 14:57:13.854255 6254 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:13.854467 6254 heartbeater.cc:511] Master 127.20.116.190:34017 requested a full tablet report, sending...
I20260928 14:57:13.855088 6066 ts_manager.cc:194] Registered new tserver with Master: fa3bec0cceed4cfa882167476df657da (127.20.116.129:35335)
I20260928 14:57:13.855696 6066 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.116.129:47629
W20260928 14:57:13.937198 6257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:13.937422 6257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:13.937446 6257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:13.939006 6257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:13.939056 6257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:13.939085 6257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:57:13.940568 6257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:13.940881 6257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:13.941228 6257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:13.946486 6262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:13.946589 6263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:13.946660 6257 server_base.cc:1034] running on GCE node
W20260928 14:57:13.946486 6265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:13.946982 6257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:13.947239 6257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:13.948438 6257 hybrid_clock.cc:648] HybridClock initialized: now 1790607433948416 us; error 37 us; skew 500 ppm
I20260928 14:57:13.949764 6257 webserver.cc:460] Webserver started at http://127.20.116.130:43689/ using document root <none> and password file <none>
I20260928 14:57:13.950008 6257 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:13.950079 6257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:13.950240 6257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:13.951428 6257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "9bd99c05c2b14d19a34f1d6598a62c99"
format_stamp: "Formatted at 2026-09-28 14:57:13 on dist-test-slave-pxl6"
server_key: "65eeb10af9694668d1bd409a19b9d926"
server_key_iv: "ef6fbdc6b83fb81a4cdd6712695c31d8"
server_key_version: "encryptionkey@0"
I20260928 14:57:13.951795 6257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "9bd99c05c2b14d19a34f1d6598a62c99"
format_stamp: "Formatted at 2026-09-28 14:57:13 on dist-test-slave-pxl6"
server_key: "65eeb10af9694668d1bd409a19b9d926"
server_key_iv: "ef6fbdc6b83fb81a4cdd6712695c31d8"
server_key_version: "encryptionkey@0"
I20260928 14:57:13.953154 6257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:57:13.953963 6271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:13.954154 6257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:57:13.954257 6257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "9bd99c05c2b14d19a34f1d6598a62c99"
format_stamp: "Formatted at 2026-09-28 14:57:13 on dist-test-slave-pxl6"
server_key: "65eeb10af9694668d1bd409a19b9d926"
server_key_iv: "ef6fbdc6b83fb81a4cdd6712695c31d8"
server_key_version: "encryptionkey@0"
I20260928 14:57:13.954342 6257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:13.976967 6257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:13.977324 6257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:13.977479 6257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:13.977736 6257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:13.978119 6257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:13.978178 6257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:13.978223 6257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:13.978248 6257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:13.984215 6257 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:39195
I20260928 14:57:13.984263 6384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:39195 every 8 connection(s)
I20260928 14:57:13.984607 6257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:57:13.988178 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 6257
I20260928 14:57:13.988287 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:57:13.988511 20946 external_mini_cluster.cc:1428] Setting key 4fc49b20d3436c42fb976ab03393f30c
I20260928 14:57:13.989480 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34017
--builtin_ntp_servers=127.20.116.148:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:57:13.990321 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.985362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:50935 (local address 127.20.116.190:34017)
0928 14:57:13.985416 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:13.985419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:13.985657 (+ 238us) server_negotiation.cc:407] Connection header received
0928 14:57:13.986155 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:13.986156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:13.986166 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:13.986223 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:13.986970 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.987466 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.988794 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.988994 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.989073 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.990036 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:13.990038 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:13.990038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:13.990043 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:13.990058 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:13.990062 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:13.990115 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:13.990214 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 14:57:13.990241 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:57:13.990597 6387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:13.985330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34017 (local address 127.20.116.130:50935)
0928 14:57:13.985524 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:13.985566 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:13.986069 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:13.986288 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:13.986292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:13.986301 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:13.986889 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.986894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.987550 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.987551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.988708 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:13.988711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:13.989087 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:13.989088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:13.989255 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:13.989875 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:57:13.989875 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:13.989904 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:57:13.989928 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:13.990120 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:13.990126 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:13.990130 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:57:13.990217 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:57:13.990337 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20260928 14:57:13.991430 6385 heartbeater.cc:347] Connected to a master server at 127.20.116.190:34017
I20260928 14:57:13.991540 6385 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:13.991741 6385 heartbeater.cc:511] Master 127.20.116.190:34017 requested a full tablet report, sending...
I20260928 14:57:13.992177 6066 ts_manager.cc:194] Registered new tserver with Master: 9bd99c05c2b14d19a34f1d6598a62c99 (127.20.116.130:39195)
I20260928 14:57:13.992787 6066 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.116.130:50935
W20260928 14:57:14.075804 6388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:14.076026 6388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:14.076048 6388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:14.077162 6388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:14.077224 6388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:14.077250 6388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:57:14.078567 6388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:34017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:14.078775 6388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:14.079025 6388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:14.084362 6394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:14.084362 6393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:14.084421 6388 server_base.cc:1034] running on GCE node
W20260928 14:57:14.084365 6396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:14.084815 6388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:14.085127 6388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:14.086293 6388 hybrid_clock.cc:648] HybridClock initialized: now 1790607434086271 us; error 40 us; skew 500 ppm
I20260928 14:57:14.087651 6388 webserver.cc:460] Webserver started at http://127.20.116.131:40267/ using document root <none> and password file <none>
I20260928 14:57:14.087917 6388 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:14.087987 6388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:14.088145 6388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:14.089303 6388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "95f321029069496682f4715fe5a21345"
format_stamp: "Formatted at 2026-09-28 14:57:14 on dist-test-slave-pxl6"
server_key: "34bdc01adc336bd370847d2e9902603e"
server_key_iv: "871f8a832adfb056943b922de3a76004"
server_key_version: "encryptionkey@0"
I20260928 14:57:14.089672 6388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "95f321029069496682f4715fe5a21345"
format_stamp: "Formatted at 2026-09-28 14:57:14 on dist-test-slave-pxl6"
server_key: "34bdc01adc336bd370847d2e9902603e"
server_key_iv: "871f8a832adfb056943b922de3a76004"
server_key_version: "encryptionkey@0"
I20260928 14:57:14.091073 6388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:57:14.091881 6402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:14.092082 6388 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:57:14.092188 6388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "95f321029069496682f4715fe5a21345"
format_stamp: "Formatted at 2026-09-28 14:57:14 on dist-test-slave-pxl6"
server_key: "34bdc01adc336bd370847d2e9902603e"
server_key_iv: "871f8a832adfb056943b922de3a76004"
server_key_version: "encryptionkey@0"
I20260928 14:57:14.092271 6388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:14.111169 6388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:14.111541 6388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:14.111701 6388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:14.111970 6388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:14.112335 6388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:14.112389 6388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:14.112433 6388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:14.112461 6388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:14.118438 6388 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:44019
I20260928 14:57:14.118511 6515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:44019 every 8 connection(s)
I20260928 14:57:14.118918 6388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:57:14.124600 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.119702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:32953 (local address 127.20.116.190:34017)
0928 14:57:14.119784 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:14.119786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:14.120078 (+ 292us) server_negotiation.cc:407] Connection header received
0928 14:57:14.120722 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:14.120724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:14.120738 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:14.120812 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:14.121539 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.122056 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.123146 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.123350 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.123433 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:14.124275 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:14.124276 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:14.124277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:14.124282 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:14.124298 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:14.124305 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:14.124372 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:14.124471 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 14:57:14.124502 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:57:14.124818 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.119683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:34017 (local address 127.20.116.131:32953)
0928 14:57:14.119943 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:14.119981 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:14.120567 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:14.120898 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:14.120902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:14.120911 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:14.121424 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:14.121429 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.122173 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.122174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:14.123033 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:14.123035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.123442 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.123444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:14.123589 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:14.124118 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:57:14.124118 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:14.124147 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:57:14.124171 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:14.124371 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:14.124379 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:14.124382 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:57:14.124474 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 14:57:14.124581 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":96,"threads_started":1}
I20260928 14:57:14.125247 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 6388
I20260928 14:57:14.125352 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:57:14.125335 6516 heartbeater.cc:347] Connected to a master server at 127.20.116.190:34017
I20260928 14:57:14.125447 6516 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:14.125589 20946 external_mini_cluster.cc:1428] Setting key 1e97ea30f61941f95aae5704b3284a14
I20260928 14:57:14.125653 6516 heartbeater.cc:511] Master 127.20.116.190:34017 requested a full tablet report, sending...
I20260928 14:57:14.126050 6065 ts_manager.cc:194] Registered new tserver with Master: 95f321029069496682f4715fe5a21345 (127.20.116.131:44019)
I20260928 14:57:14.126566 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:57:14.126578 6065 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.116.131:32953
I20260928 14:57:14.130684 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.127979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60500 (local address 127.20.116.190:34017)
0928 14:57:14.128020 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:14.128022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:14.128163 (+ 141us) server_negotiation.cc:407] Connection header received
0928 14:57:14.128188 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:14.128189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:14.128197 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:14.128250 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:14.128380 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.128828 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.129574 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.129783 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.129850 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:14.130426 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:14.130427 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:14.130427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:14.130431 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:14.130442 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:14.130445 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:14.130485 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:14.130519 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 14:57:14.130596 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:57:14.132360 6065 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:57:14.133886 6065 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:57:14.140419 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.136322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43968 (local address 127.20.116.131:44019)
0928 14:57:14.136651 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:14.136656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:14.136669 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:57:14.136712 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:14.136714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:14.136732 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:14.136825 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:14.137022 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.138211 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.139508 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.140007 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.140107 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:14.140204 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:14.140292 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 14:57:14.140329 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":80,"threads_started":1}
I20260928 14:57:14.140407 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.136142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:44019 (local address 127.0.0.1:43968)
0928 14:57:14.136261 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:14.136303 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:14.136374 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:14.136846 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:14.136850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:14.136853 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:14.136933 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:14.136936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.138319 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.138320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:14.139439 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:14.139441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.140109 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.140110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:14.140182 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:14.140195 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 14:57:14.140278 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 14:57:14.140331 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
I20260928 14:57:14.140640 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.136143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:35335 (local address 127.0.0.1:50546)
0928 14:57:14.136406 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:14.136417 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:14.136471 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:14.136887 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:14.136888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:14.136891 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:14.136980 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:14.136982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.138036 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.138038 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:14.139358 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:14.139361 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.140282 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.140283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:14.140345 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:14.140350 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:57:14.140382 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 14:57:14.140423 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":81,"threads_started":1}
I20260928 14:57:14.140729 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.136508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50546 (local address 127.20.116.129:35335)
0928 14:57:14.136715 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:14.136727 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:14.136736 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:57:14.136773 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:14.136775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:14.136789 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:14.137130 (+ 341us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:14.137197 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.137947 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.139450 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.140168 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.140472 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:14.140569 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:14.140594 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 14:57:14.140637 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":105,"threads_started":1}
I20260928 14:57:14.140973 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.136143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:39195 (local address 127.0.0.1:46974)
0928 14:57:14.136927 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:14.136938 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:14.136985 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:14.137253 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:14.137254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:14.137256 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:14.137290 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:14.137291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.138623 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.138624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:14.139670 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:14.139672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.140659 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.140660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:14.140815 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:14.140819 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:57:14.140865 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 14:57:14.140902 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260928 14:57:14.141294 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.136399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.20.116.130:39195)
0928 14:57:14.136592 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:14.136597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:14.137047 (+ 450us) server_negotiation.cc:407] Connection header received
0928 14:57:14.137092 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:14.137095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:14.137110 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:14.137175 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:14.137372 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.138513 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.139772 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.140493 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.140552 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:14.140624 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:14.141171 (+ 547us) server_negotiation.cc:299] Negotiation successful
0928 14:57:14.141215 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260928 14:57:14.141512 6450 tablet_service.cc:1467] Processing CreateTablet for tablet dded0defc4824372b6a48d9cf4455d2b (DEFAULT_TABLE table=test-table [id=3fa81100031d49f4a4f252f8fa46e972]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:14.141505 6188 tablet_service.cc:1467] Processing CreateTablet for tablet dded0defc4824372b6a48d9cf4455d2b (DEFAULT_TABLE table=test-table [id=3fa81100031d49f4a4f252f8fa46e972]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:14.141844 6450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dded0defc4824372b6a48d9cf4455d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:14.141853 6188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dded0defc4824372b6a48d9cf4455d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:14.141939 6319 tablet_service.cc:1467] Processing CreateTablet for tablet dded0defc4824372b6a48d9cf4455d2b (DEFAULT_TABLE table=test-table [id=3fa81100031d49f4a4f252f8fa46e972]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:14.142230 6319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dded0defc4824372b6a48d9cf4455d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:14.144541 6530 tablet_bootstrap.cc:492] T dded0defc4824372b6a48d9cf4455d2b P 95f321029069496682f4715fe5a21345: Bootstrap starting.
I20260928 14:57:14.145207 6530 tablet_bootstrap.cc:654] T dded0defc4824372b6a48d9cf4455d2b P 95f321029069496682f4715fe5a21345: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:14.145259 6531 tablet_bootstrap.cc:492] T dded0defc4824372b6a48d9cf4455d2b P fa3bec0cceed4cfa882167476df657da: Bootstrap starting.
I20260928 14:57:14.145485 6532 tablet_bootstrap.cc:492] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99: Bootstrap starting.
I20260928 14:57:14.145735 6530 log.cc:824] T dded0defc4824372b6a48d9cf4455d2b P 95f321029069496682f4715fe5a21345: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:14.145975 6531 tablet_bootstrap.cc:654] T dded0defc4824372b6a48d9cf4455d2b P fa3bec0cceed4cfa882167476df657da: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:14.146270 6532 tablet_bootstrap.cc:654] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:14.146313 6531 log.cc:824] T dded0defc4824372b6a48d9cf4455d2b P fa3bec0cceed4cfa882167476df657da: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:14.146629 6532 log.cc:824] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:14.146646 6530 tablet_bootstrap.cc:492] T dded0defc4824372b6a48d9cf4455d2b P 95f321029069496682f4715fe5a21345: No bootstrap required, opened a new log
I20260928 14:57:14.146739 6530 ts_tablet_manager.cc:1397] T dded0defc4824372b6a48d9cf4455d2b P 95f321029069496682f4715fe5a21345: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:57:14.147122 6531 tablet_bootstrap.cc:492] T dded0defc4824372b6a48d9cf4455d2b P fa3bec0cceed4cfa882167476df657da: No bootstrap required, opened a new log
I20260928 14:57:14.147195 6531 ts_tablet_manager.cc:1397] T dded0defc4824372b6a48d9cf4455d2b P fa3bec0cceed4cfa882167476df657da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:57:14.149456 6530 raft_consensus.cc:348] T dded0defc4824372b6a48d9cf4455d2b P 95f321029069496682f4715fe5a21345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f321029069496682f4715fe5a21345" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 44019 } } peers { permanent_uuid: "fa3bec0cceed4cfa882167476df657da" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35335 } } peers { permanent_uuid: "9bd99c05c2b14d19a34f1d6598a62c99" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 39195 } }
I20260928 14:57:14.149632 6530 raft_consensus.cc:374] T dded0defc4824372b6a48d9cf4455d2b P 95f321029069496682f4715fe5a21345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:14.149672 6530 raft_consensus.cc:729] T dded0defc4824372b6a48d9cf4455d2b P 95f321029069496682f4715fe5a21345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f321029069496682f4715fe5a21345, State: Initialized, Role: FOLLOWER
I20260928 14:57:14.149701 6531 raft_consensus.cc:348] T dded0defc4824372b6a48d9cf4455d2b P fa3bec0cceed4cfa882167476df657da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f321029069496682f4715fe5a21345" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 44019 } } peers { permanent_uuid: "fa3bec0cceed4cfa882167476df657da" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35335 } } peers { permanent_uuid: "9bd99c05c2b14d19a34f1d6598a62c99" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 39195 } }
I20260928 14:57:14.149798 6530 consensus_queue.cc:260] T dded0defc4824372b6a48d9cf4455d2b P 95f321029069496682f4715fe5a21345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f321029069496682f4715fe5a21345" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 44019 } } peers { permanent_uuid: "fa3bec0cceed4cfa882167476df657da" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35335 } } peers { permanent_uuid: "9bd99c05c2b14d19a34f1d6598a62c99" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 39195 } }
I20260928 14:57:14.149852 6531 raft_consensus.cc:374] T dded0defc4824372b6a48d9cf4455d2b P fa3bec0cceed4cfa882167476df657da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:14.149886 6531 raft_consensus.cc:729] T dded0defc4824372b6a48d9cf4455d2b P fa3bec0cceed4cfa882167476df657da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa3bec0cceed4cfa882167476df657da, State: Initialized, Role: FOLLOWER
I20260928 14:57:14.150007 6531 consensus_queue.cc:260] T dded0defc4824372b6a48d9cf4455d2b P fa3bec0cceed4cfa882167476df657da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f321029069496682f4715fe5a21345" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 44019 } } peers { permanent_uuid: "fa3bec0cceed4cfa882167476df657da" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35335 } } peers { permanent_uuid: "9bd99c05c2b14d19a34f1d6598a62c99" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 39195 } }
I20260928 14:57:14.150059 6530 ts_tablet_manager.cc:1428] T dded0defc4824372b6a48d9cf4455d2b P 95f321029069496682f4715fe5a21345: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:57:14.150097 6516 heartbeater.cc:503] Master 127.20.116.190:34017 was elected leader, sending a full tablet report...
I20260928 14:57:14.150259 6531 ts_tablet_manager.cc:1428] T dded0defc4824372b6a48d9cf4455d2b P fa3bec0cceed4cfa882167476df657da: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 14:57:14.150394 6254 heartbeater.cc:503] Master 127.20.116.190:34017 was elected leader, sending a full tablet report...
I20260928 14:57:14.150763 6532 tablet_bootstrap.cc:492] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99: No bootstrap required, opened a new log
I20260928 14:57:14.150840 6532 ts_tablet_manager.cc:1397] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260928 14:57:14.152853 6532 raft_consensus.cc:348] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f321029069496682f4715fe5a21345" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 44019 } } peers { permanent_uuid: "fa3bec0cceed4cfa882167476df657da" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35335 } } peers { permanent_uuid: "9bd99c05c2b14d19a34f1d6598a62c99" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 39195 } }
I20260928 14:57:14.152997 6532 raft_consensus.cc:374] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:14.153018 6532 raft_consensus.cc:729] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd99c05c2b14d19a34f1d6598a62c99, State: Initialized, Role: FOLLOWER
I20260928 14:57:14.153141 6532 consensus_queue.cc:260] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f321029069496682f4715fe5a21345" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 44019 } } peers { permanent_uuid: "fa3bec0cceed4cfa882167476df657da" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35335 } } peers { permanent_uuid: "9bd99c05c2b14d19a34f1d6598a62c99" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 39195 } }
I20260928 14:57:14.153443 6532 ts_tablet_manager.cc:1428] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260928 14:57:14.153475 6385 heartbeater.cc:503] Master 127.20.116.190:34017 was elected leader, sending a full tablet report...
W20260928 14:57:14.235240 6386 tablet.cc:2369] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:57:14.347847 6255 tablet.cc:2369] T dded0defc4824372b6a48d9cf4455d2b P fa3bec0cceed4cfa882167476df657da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:57:14.369520 6517 tablet.cc:2369] T dded0defc4824372b6a48d9cf4455d2b P 95f321029069496682f4715fe5a21345: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:57:14.408138 6538 raft_consensus.cc:482] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:57:14.408293 6538 raft_consensus.cc:504] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f321029069496682f4715fe5a21345" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 44019 } } peers { permanent_uuid: "fa3bec0cceed4cfa882167476df657da" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35335 } } peers { permanent_uuid: "9bd99c05c2b14d19a34f1d6598a62c99" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 39195 } }
I20260928 14:57:14.408622 6538 leader_election.cc:288] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95f321029069496682f4715fe5a21345 (127.20.116.131:44019), fa3bec0cceed4cfa882167476df657da (127.20.116.129:35335)
I20260928 14:57:14.412766 6387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.408867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:35335 (local address 127.20.116.130:41213)
0928 14:57:14.408939 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:14.408952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:14.409002 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:14.409213 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:14.409214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:14.409216 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:14.409339 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:14.409342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.410325 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.410326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:14.411701 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:14.411705 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.412452 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.412453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:14.412542 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:14.412549 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 14:57:14.412616 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 14:57:14.412671 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":62}
I20260928 14:57:14.412773 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.408868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:44019 (local address 127.20.116.130:42019)
0928 14:57:14.409072 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:14.409083 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:14.409134 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:14.409300 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:14.409302 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:14.409303 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:14.409357 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:14.409359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.410325 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.410326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:14.411715 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:14.411717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.412452 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.412453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:14.412542 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:14.412555 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 14:57:14.412663 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 14:57:14.412693 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":89,"thread_start_us":117,"threads_started":1}
I20260928 14:57:14.412834 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.408962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:41213 (local address 127.20.116.129:35335)
0928 14:57:14.409019 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:14.409021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:14.409073 (+ 52us) server_negotiation.cc:407] Connection header received
0928 14:57:14.409100 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:14.409100 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:14.409109 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:14.409187 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:14.409444 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.410199 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.411808 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.412346 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.412430 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:14.412440 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:14.412706 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 14:57:14.412759 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 14:57:14.412834 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.408962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:42019 (local address 127.20.116.131:44019)
0928 14:57:14.409024 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:14.409026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:14.409152 (+ 126us) server_negotiation.cc:407] Connection header received
0928 14:57:14.409183 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:14.409184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:14.409193 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:14.409278 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:14.409436 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.410181 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.411807 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.412369 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.412432 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:14.412440 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:14.412706 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 14:57:14.412759 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:57:14.413163 6208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dded0defc4824372b6a48d9cf4455d2b" candidate_uuid: "9bd99c05c2b14d19a34f1d6598a62c99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa3bec0cceed4cfa882167476df657da" is_pre_election: true
I20260928 14:57:14.413187 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dded0defc4824372b6a48d9cf4455d2b" candidate_uuid: "9bd99c05c2b14d19a34f1d6598a62c99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95f321029069496682f4715fe5a21345" is_pre_election: true
I20260928 14:57:14.413340 6208 raft_consensus.cc:2454] T dded0defc4824372b6a48d9cf4455d2b P fa3bec0cceed4cfa882167476df657da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bd99c05c2b14d19a34f1d6598a62c99 in term 0.
I20260928 14:57:14.413362 6470 raft_consensus.cc:2454] T dded0defc4824372b6a48d9cf4455d2b P 95f321029069496682f4715fe5a21345 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bd99c05c2b14d19a34f1d6598a62c99 in term 0.
I20260928 14:57:14.413628 6274 leader_election.cc:302] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95f321029069496682f4715fe5a21345, 9bd99c05c2b14d19a34f1d6598a62c99; no voters:
I20260928 14:57:14.413779 6538 raft_consensus.cc:2781] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:57:14.413846 6538 raft_consensus.cc:482] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:57:14.413863 6538 raft_consensus.cc:3037] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:14.414664 6538 raft_consensus.cc:504] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f321029069496682f4715fe5a21345" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 44019 } } peers { permanent_uuid: "fa3bec0cceed4cfa882167476df657da" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35335 } } peers { permanent_uuid: "9bd99c05c2b14d19a34f1d6598a62c99" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 39195 } }
I20260928 14:57:14.414824 6538 leader_election.cc:288] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99 [CANDIDATE]: Term 1 election: Requested vote from peers 95f321029069496682f4715fe5a21345 (127.20.116.131:44019), fa3bec0cceed4cfa882167476df657da (127.20.116.129:35335)
I20260928 14:57:14.414996 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dded0defc4824372b6a48d9cf4455d2b" candidate_uuid: "9bd99c05c2b14d19a34f1d6598a62c99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95f321029069496682f4715fe5a21345"
I20260928 14:57:14.415007 6208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dded0defc4824372b6a48d9cf4455d2b" candidate_uuid: "9bd99c05c2b14d19a34f1d6598a62c99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa3bec0cceed4cfa882167476df657da"
I20260928 14:57:14.415100 6208 raft_consensus.cc:3037] T dded0defc4824372b6a48d9cf4455d2b P fa3bec0cceed4cfa882167476df657da [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:14.415074 6470 raft_consensus.cc:3037] T dded0defc4824372b6a48d9cf4455d2b P 95f321029069496682f4715fe5a21345 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:14.415833 6208 raft_consensus.cc:2454] T dded0defc4824372b6a48d9cf4455d2b P fa3bec0cceed4cfa882167476df657da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bd99c05c2b14d19a34f1d6598a62c99 in term 1.
I20260928 14:57:14.415903 6470 raft_consensus.cc:2454] T dded0defc4824372b6a48d9cf4455d2b P 95f321029069496682f4715fe5a21345 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bd99c05c2b14d19a34f1d6598a62c99 in term 1.
I20260928 14:57:14.416069 6273 leader_election.cc:302] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bd99c05c2b14d19a34f1d6598a62c99, fa3bec0cceed4cfa882167476df657da; no voters:
I20260928 14:57:14.416157 6538 raft_consensus.cc:2781] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:14.416308 6538 raft_consensus.cc:686] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99 [term 1 LEADER]: Becoming Leader. State: Replica: 9bd99c05c2b14d19a34f1d6598a62c99, State: Running, Role: LEADER
I20260928 14:57:14.416420 6538 consensus_queue.cc:237] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f321029069496682f4715fe5a21345" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 44019 } } peers { permanent_uuid: "fa3bec0cceed4cfa882167476df657da" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35335 } } peers { permanent_uuid: "9bd99c05c2b14d19a34f1d6598a62c99" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 39195 } }
I20260928 14:57:14.417343 6065 catalog_manager.cc:5495] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bd99c05c2b14d19a34f1d6598a62c99 (127.20.116.130). New cstate: current_term: 1 leader_uuid: "9bd99c05c2b14d19a34f1d6598a62c99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f321029069496682f4715fe5a21345" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 44019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa3bec0cceed4cfa882167476df657da" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bd99c05c2b14d19a34f1d6598a62c99" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 39195 } health_report { overall_health: HEALTHY } } }
I20260928 14:57:14.424814 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.422287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46976 (local address 127.20.116.130:39195)
0928 14:57:14.422352 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:14.422354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:14.422361 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:57:14.422394 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:14.422395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:14.422420 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:14.422482 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:14.422671 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.423126 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.423907 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:14.424107 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:14.424171 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:14.424345 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:14.424451 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:14.424521 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:14.424556 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 14:57:14.424575 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:57:14.426769 6208 raft_consensus.cc:1261] T dded0defc4824372b6a48d9cf4455d2b P fa3bec0cceed4cfa882167476df657da [term 1 FOLLOWER]: Refusing update from remote peer 9bd99c05c2b14d19a34f1d6598a62c99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:14.426795 6470 raft_consensus.cc:1261] T dded0defc4824372b6a48d9cf4455d2b P 95f321029069496682f4715fe5a21345 [term 1 FOLLOWER]: Refusing update from remote peer 9bd99c05c2b14d19a34f1d6598a62c99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:14.427163 6538 consensus_queue.cc:1035] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f321029069496682f4715fe5a21345" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:14.427390 6540 consensus_queue.cc:1035] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa3bec0cceed4cfa882167476df657da" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 35335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:14.429744 6543 mvcc.cc:205] Tried to move back new op lower bound from 7334328051408547840 to 7334328051370373120. Current Snapshot: MvccSnapshot[applied={T|T < 7334328051408547840}]
I20260928 14:57:14.429946 6544 mvcc.cc:205] Tried to move back new op lower bound from 7334328051408547840 to 7334328051370373120. Current Snapshot: MvccSnapshot[applied={T|T < 7334328051408547840}]
I20260928 14:57:14.432621 6064 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:14.432734 6064 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:14.433544 6064 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54aa532a740d45ff9d91532325ed6730: Sending DeleteTablet for 3 replicas of tablet dded0defc4824372b6a48d9cf4455d2b
I20260928 14:57:14.433881 6188 tablet_service.cc:1514] Processing DeleteTablet for tablet dded0defc4824372b6a48d9cf4455d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:14 UTC) from {username='slave'} at 127.0.0.1:50546
I20260928 14:57:14.433894 6450 tablet_service.cc:1514] Processing DeleteTablet for tablet dded0defc4824372b6a48d9cf4455d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:14 UTC) from {username='slave'} at 127.0.0.1:43968
I20260928 14:57:14.434103 6319 tablet_service.cc:1514] Processing DeleteTablet for tablet dded0defc4824372b6a48d9cf4455d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:14 UTC) from {username='slave'} at 127.0.0.1:46974
I20260928 14:57:14.434135 6552 tablet_replica.cc:330] T dded0defc4824372b6a48d9cf4455d2b P fa3bec0cceed4cfa882167476df657da: stopping tablet replica
I20260928 14:57:14.434240 6552 raft_consensus.cc:2229] T dded0defc4824372b6a48d9cf4455d2b P fa3bec0cceed4cfa882167476df657da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:14.434295 6554 tablet_replica.cc:330] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99: stopping tablet replica
I20260928 14:57:14.434424 6554 raft_consensus.cc:2229] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:57:14.434453 6553 tablet_replica.cc:330] T dded0defc4824372b6a48d9cf4455d2b P 95f321029069496682f4715fe5a21345: stopping tablet replica
I20260928 14:57:14.434583 6554 raft_consensus.cc:2258] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:14.434585 6553 raft_consensus.cc:2229] T dded0defc4824372b6a48d9cf4455d2b P 95f321029069496682f4715fe5a21345 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:14.434726 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 6126
I20260928 14:57:14.435554 6553 raft_consensus.cc:2258] T dded0defc4824372b6a48d9cf4455d2b P 95f321029069496682f4715fe5a21345 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:14.435981 6553 ts_tablet_manager.cc:1905] T dded0defc4824372b6a48d9cf4455d2b P 95f321029069496682f4715fe5a21345: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:14.437613 6553 ts_tablet_manager.cc:1918] T dded0defc4824372b6a48d9cf4455d2b P 95f321029069496682f4715fe5a21345: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:14.437700 6553 log.cc:1196] T dded0defc4824372b6a48d9cf4455d2b P 95f321029069496682f4715fe5a21345: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/dded0defc4824372b6a48d9cf4455d2b
I20260928 14:57:14.438030 6553 ts_tablet_manager.cc:1939] T dded0defc4824372b6a48d9cf4455d2b P 95f321029069496682f4715fe5a21345: Deleting consensus metadata
I20260928 14:57:14.438459 6052 catalog_manager.cc:4841] TS 95f321029069496682f4715fe5a21345 (127.20.116.131:44019): tablet dded0defc4824372b6a48d9cf4455d2b (table test-table [id=3fa81100031d49f4a4f252f8fa46e972]) successfully deleted
I20260928 14:57:14.439284 6554 ts_tablet_manager.cc:1905] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:14.440587 6554 ts_tablet_manager.cc:1918] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:14.440675 6554 log.cc:1196] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/dded0defc4824372b6a48d9cf4455d2b
I20260928 14:57:14.440971 6554 ts_tablet_manager.cc:1939] T dded0defc4824372b6a48d9cf4455d2b P 9bd99c05c2b14d19a34f1d6598a62c99: Deleting consensus metadata
I20260928 14:57:14.441342 6050 catalog_manager.cc:4841] TS 9bd99c05c2b14d19a34f1d6598a62c99 (127.20.116.130:39195): tablet dded0defc4824372b6a48d9cf4455d2b (table test-table [id=3fa81100031d49f4a4f252f8fa46e972]) successfully deleted
W20260928 14:57:14.443539 6051 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.116.129:35335 (error 108)
I20260928 14:57:14.443712 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 6257
I20260928 14:57:14.446893 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:14.446714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:35335 (local address 127.0.0.1:50556)
0928 14:57:14.446785 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:14.446849 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.129:35335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 14:57:14.447057 6051 catalog_manager.cc:4584] TS fa3bec0cceed4cfa882167476df657da (127.20.116.129:35335): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dded0defc4824372b6a48d9cf4455d2b: Network error: Client connection negotiation failed: client connection to 127.20.116.129:35335: connect: Connection refused (error 111)
I20260928 14:57:14.449792 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 6388
I20260928 14:57:14.454707 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 6035
2026-09-28T14:57:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (943 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:14 dist-test-slave-pxl6 krb5kdc[6558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:14 dist-test-slave-pxl6 krb5kdc[6558](info): set up 2 sockets
Sep 28 14:57:14 dist-test-slave-pxl6 krb5kdc[6558](info): commencing operation
krb5kdc: starting...
W20260928 14:57:15.491412 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:15 dist-test-slave-pxl6 krb5kdc[6558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:57:15.613556 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:36501
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:36501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:15.694926 6571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:15.695178 6571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:15.695215 6571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:15.696373 6571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:15.696447 6571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:15.696473 6571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:15.696506 6571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:15.696528 6571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:15.698069 6571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:36501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:36501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:15.698354 6571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:15.698643 6571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:15.703878 6579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:15.703882 6577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:15.703878 6576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:15.703949 6571 server_base.cc:1034] running on GCE node
I20260928 14:57:15.704439 6571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:15.704757 6571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:15.705937 6571 hybrid_clock.cc:648] HybridClock initialized: now 1790607435705922 us; error 41 us; skew 500 ppm
Sep 28 14:57:15 dist-test-slave-pxl6 krb5kdc[6558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607435, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:15.709077 6571 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:57:15.709728 6571 webserver.cc:460] Webserver started at http://127.20.116.190:40905/ using document root <none> and password file <none>
I20260928 14:57:15.710019 6571 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:15.710091 6571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:15.710258 6571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:15.711901 6571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "759dcb9340f34cc4947af6e53a8b95ae"
format_stamp: "Formatted at 2026-09-28 14:57:15 on dist-test-slave-pxl6"
server_key: "9027999f793ce143628af530a4086180"
server_key_iv: "ca8f2b68f9d831cd8a70e8b02adf5e4d"
server_key_version: "encryptionkey@0"
I20260928 14:57:15.712322 6571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "759dcb9340f34cc4947af6e53a8b95ae"
format_stamp: "Formatted at 2026-09-28 14:57:15 on dist-test-slave-pxl6"
server_key: "9027999f793ce143628af530a4086180"
server_key_iv: "ca8f2b68f9d831cd8a70e8b02adf5e4d"
server_key_version: "encryptionkey@0"
I20260928 14:57:15.713990 6571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:15.714941 6586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:15.715144 6571 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:57:15.715252 6571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "759dcb9340f34cc4947af6e53a8b95ae"
format_stamp: "Formatted at 2026-09-28 14:57:15 on dist-test-slave-pxl6"
server_key: "9027999f793ce143628af530a4086180"
server_key_iv: "ca8f2b68f9d831cd8a70e8b02adf5e4d"
server_key_version: "encryptionkey@0"
I20260928 14:57:15.715322 6571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:15.734956 6571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:15.737038 6571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:15.737229 6571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:15.741310 6571 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:36501
I20260928 14:57:15.741361 6638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:36501 every 8 connection(s)
I20260928 14:57:15.741739 6571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:15.742738 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:15.745050 6639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae: Bootstrap starting.
I20260928 14:57:15.745651 6639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:15.745962 6639 log.cc:824] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:15.746706 6639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae: No bootstrap required, opened a new log
I20260928 14:57:15.748209 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 6571
I20260928 14:57:15.748344 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:15.748541 20946 external_mini_cluster.cc:1428] Setting key ba0db3b55316cb6948a0df1a8e224baa
I20260928 14:57:15.748438 6639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759dcb9340f34cc4947af6e53a8b95ae" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36501 } }
I20260928 14:57:15.748582 6639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:15.748637 6639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759dcb9340f34cc4947af6e53a8b95ae, State: Initialized, Role: FOLLOWER
I20260928 14:57:15.748826 6639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759dcb9340f34cc4947af6e53a8b95ae" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36501 } }
I20260928 14:57:15.748951 6639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:15.748992 6639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:15.749038 6639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:15.749681 6639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759dcb9340f34cc4947af6e53a8b95ae" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36501 } }
I20260928 14:57:15.749801 6639 leader_election.cc:302] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 759dcb9340f34cc4947af6e53a8b95ae; no voters:
I20260928 14:57:15.749971 6639 leader_election.cc:288] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:15.750008 6644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:15.750156 6644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae [term 1 LEADER]: Becoming Leader. State: Replica: 759dcb9340f34cc4947af6e53a8b95ae, State: Running, Role: LEADER
I20260928 14:57:15.750249 6639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:15.750259 6644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759dcb9340f34cc4947af6e53a8b95ae" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36501 } }
I20260928 14:57:15.750662 6644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 759dcb9340f34cc4947af6e53a8b95ae. Latest consensus state: current_term: 1 leader_uuid: "759dcb9340f34cc4947af6e53a8b95ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759dcb9340f34cc4947af6e53a8b95ae" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36501 } } }
I20260928 14:57:15.750748 6644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:15.750667 6646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "759dcb9340f34cc4947af6e53a8b95ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759dcb9340f34cc4947af6e53a8b95ae" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36501 } } }
I20260928 14:57:15.750913 6646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:15.751041 6650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:15.751610 6650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:15.754827 6650 catalog_manager.cc:1347] Generated new cluster ID: 653defe8e2f342e29d17794b7d4edc32
I20260928 14:57:15.754887 6650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:57:15 dist-test-slave-pxl6 krb5kdc[6558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:57:15.759764 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:15.749091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58004 (local address 127.20.116.190:36501)
0928 14:57:15.749300 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:15.749311 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:15.749346 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:57:15.749683 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:15.749689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:15.749741 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:15.749890 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:15.750808 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:15.752318 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:15.753259 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:15.753897 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:15.754013 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:15.756477 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:15.756486 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:15.756487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:15.756498 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:15.758183 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:15.758586 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:15.758589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:15.758590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:15.758688 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:15.758895 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:15.758896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:15.758896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:15.759142 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:15.759343 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:15.759451 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 14:57:15.759594 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":105,"threads_started":1}
I20260928 14:57:15.771741 6650 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:15.772315 6650 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:15.774493 6650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae: Generated new TSK 0
I20260928 14:57:15.774788 6650 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
I20260928 14:57:15.806739 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:36501
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:15.907406 6665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:15.907666 6665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:15.907692 6665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:15.908891 6665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:15.908933 6665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:15.908960 6665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:57:15.910478 6665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:36501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:15.910766 6665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:15.911008 6665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:15.916441 6673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:15.916428 6670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:15.916431 6671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:15.916868 6665 server_base.cc:1034] running on GCE node
I20260928 14:57:15.917122 6665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:15.917399 6665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:15.918613 6665 hybrid_clock.cc:648] HybridClock initialized: now 1790607435918557 us; error 67 us; skew 500 ppm
Sep 28 14:57:15 dist-test-slave-pxl6 krb5kdc[6558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607435, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:15.921224 6665 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:57:15.921762 6665 webserver.cc:460] Webserver started at http://127.20.116.129:33191/ using document root <none> and password file <none>
I20260928 14:57:15.921998 6665 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:15.922043 6665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:15.922205 6665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:15.923360 6665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "e2b702ae4de848629ab0258bfb6200c6"
format_stamp: "Formatted at 2026-09-28 14:57:15 on dist-test-slave-pxl6"
server_key: "61f41acbda1151eeaf2c1c15d9f4cb39"
server_key_iv: "40fd0d197eda268d856e06e345fd6268"
server_key_version: "encryptionkey@0"
I20260928 14:57:15.923671 6665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "e2b702ae4de848629ab0258bfb6200c6"
format_stamp: "Formatted at 2026-09-28 14:57:15 on dist-test-slave-pxl6"
server_key: "61f41acbda1151eeaf2c1c15d9f4cb39"
server_key_iv: "40fd0d197eda268d856e06e345fd6268"
server_key_version: "encryptionkey@0"
I20260928 14:57:15.925006 6665 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:57:15.925866 6680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:15.926077 6665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:57:15.926187 6665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "e2b702ae4de848629ab0258bfb6200c6"
format_stamp: "Formatted at 2026-09-28 14:57:15 on dist-test-slave-pxl6"
server_key: "61f41acbda1151eeaf2c1c15d9f4cb39"
server_key_iv: "40fd0d197eda268d856e06e345fd6268"
server_key_version: "encryptionkey@0"
I20260928 14:57:15.926272 6665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:15.960995 6665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:15.963172 6665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:15.963333 6665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:15.963587 6665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:15.963922 6665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:15.963972 6665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:15.964013 6665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:15.964037 6665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:15.970165 6665 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:41637
I20260928 14:57:15.970221 6793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:41637 every 8 connection(s)
I20260928 14:57:15.970549 6665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:57:15.973122 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 6665
I20260928 14:57:15.973272 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:57:15.973526 20946 external_mini_cluster.cc:1428] Setting key 4bde30e1f03b7bc48506363ff3dee113
Sep 28 14:57:15 dist-test-slave-pxl6 krb5kdc[6558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607435, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:57:15.980758 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:15.971264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:50123 (local address 127.20.116.190:36501)
0928 14:57:15.971331 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:15.971333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:15.971539 (+ 206us) server_negotiation.cc:407] Connection header received
0928 14:57:15.972085 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:15.972087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:15.972103 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:15.972178 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:15.973173 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:15.973853 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:15.975109 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:15.975349 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:15.975433 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:15.977774 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:15.977776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:15.977776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:15.977785 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:15.979351 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:15.979793 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:15.979798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:15.979798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:15.979858 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:15.980114 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:15.980115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:15.980116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:15.980229 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:15.980300 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:15.980575 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 14:57:15.980665 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:57:15.980971 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:15.971251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:36501 (local address 127.20.116.129:50123)
0928 14:57:15.971423 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:15.971460 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:15.971974 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:15.972255 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:15.972258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:15.972842 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:15.973046 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:15.973051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:15.973936 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:15.973937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:15.975021 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:15.975024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:15.975447 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:15.975448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:15.975594 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:15.976203 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:15.976212 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:15.977684 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:15.979455 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:15.979463 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:15.979466 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:15.979710 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:15.979925 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:15.979926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:15.979926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:15.980034 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:15.980292 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:15.980299 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:15.980438 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 14:57:15.980540 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 14:57:15.980712 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260928 14:57:15.981416 6794 heartbeater.cc:347] Connected to a master server at 127.20.116.190:36501
I20260928 14:57:15.981509 6794 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:15.981709 6794 heartbeater.cc:511] Master 127.20.116.190:36501 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.116.130@KRBTEST.COM; defaulting to no policy
I20260928 14:57:15.982262 6603 ts_manager.cc:194] Registered new tserver with Master: e2b702ae4de848629ab0258bfb6200c6 (127.20.116.129:41637)
I20260928 14:57:15.982949 6603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:50123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
I20260928 14:57:16.002671 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:36501
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:16.086148 6799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:16.086372 6799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:16.086397 6799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:16.087558 6799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:16.087596 6799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:16.087622 6799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:57:16.089084 6799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:36501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:16.089295 6799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:16.089483 6799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:16.094754 6807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:16.094782 6804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:16.094755 6805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:16.095057 6799 server_base.cc:1034] running on GCE node
I20260928 14:57:16.095284 6799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:16.095495 6799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:16.096674 6799 hybrid_clock.cc:648] HybridClock initialized: now 1790607436096663 us; error 33 us; skew 500 ppm
Sep 28 14:57:16 dist-test-slave-pxl6 krb5kdc[6558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607436, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:16.099447 6799 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:57:16.099964 6799 webserver.cc:460] Webserver started at http://127.20.116.130:40373/ using document root <none> and password file <none>
I20260928 14:57:16.100216 6799 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:16.100284 6799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:16.100430 6799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:16.101511 6799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "950c10dfef394e9f944c6af4280fa7bb"
format_stamp: "Formatted at 2026-09-28 14:57:16 on dist-test-slave-pxl6"
server_key: "543f67344ab1165046e58293d36f1106"
server_key_iv: "7097a89c58332ecca6403207bcc80b17"
server_key_version: "encryptionkey@0"
I20260928 14:57:16.101892 6799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "950c10dfef394e9f944c6af4280fa7bb"
format_stamp: "Formatted at 2026-09-28 14:57:16 on dist-test-slave-pxl6"
server_key: "543f67344ab1165046e58293d36f1106"
server_key_iv: "7097a89c58332ecca6403207bcc80b17"
server_key_version: "encryptionkey@0"
I20260928 14:57:16.103305 6799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 14:57:16.104084 6814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:16.104247 6799 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:57:16.104351 6799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "950c10dfef394e9f944c6af4280fa7bb"
format_stamp: "Formatted at 2026-09-28 14:57:16 on dist-test-slave-pxl6"
server_key: "543f67344ab1165046e58293d36f1106"
server_key_iv: "7097a89c58332ecca6403207bcc80b17"
server_key_version: "encryptionkey@0"
I20260928 14:57:16.104423 6799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:16.132031 6799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:16.134684 6799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:16.134855 6799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:16.135113 6799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:16.135450 6799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:16.135502 6799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:16.135545 6799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:16.135584 6799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:16.141568 6799 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:42131
I20260928 14:57:16.141616 6927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:42131 every 8 connection(s)
I20260928 14:57:16.141970 6799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:57:16.148202 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 6799
I20260928 14:57:16.148299 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
Sep 28 14:57:16 dist-test-slave-pxl6 krb5kdc[6558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607436, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:57:16.148504 20946 external_mini_cluster.cc:1428] Setting key 7e154d1e609b3c7a6ccfa8b9f9453b2c
I20260928 14:57:16.152429 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:16.142818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:44323 (local address 127.20.116.190:36501)
0928 14:57:16.142905 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:16.142908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:16.143091 (+ 183us) server_negotiation.cc:407] Connection header received
0928 14:57:16.143632 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:16.143633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:16.143645 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:16.143705 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:16.144709 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.145244 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.146290 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.146506 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.146630 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:16.149078 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:16.149079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:16.149079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:16.149088 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:16.150869 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:16.151314 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:16.151316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:16.151316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:16.151379 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:16.151704 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:16.151705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:16.151706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:16.151856 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:16.151937 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:16.152226 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 14:57:16.152334 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 14:57:16.152635 6930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:16.142764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:36501 (local address 127.20.116.130:44323)
0928 14:57:16.142958 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:16.143005 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:16.143524 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:16.143782 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:16.143787 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:16.144381 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:16.144591 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:16.144596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.145351 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.145352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:16.146191 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:16.146193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.146649 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.146650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:16.146787 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:16.147360 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:16.147369 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:16.148997 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:16.150968 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:16.150977 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:16.150979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:16.151213 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:16.151448 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:16.151449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:16.151449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:16.151631 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:16.151927 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:16.151935 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:16.152090 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 14:57:16.152194 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 14:57:16.152376 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":95,"threads_started":1}
I20260928 14:57:16.153102 6928 heartbeater.cc:347] Connected to a master server at 127.20.116.190:36501
I20260928 14:57:16.153230 6928 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:16.153465 6928 heartbeater.cc:511] Master 127.20.116.190:36501 requested a full tablet report, sending...
I20260928 14:57:16.153925 6603 ts_manager.cc:194] Registered new tserver with Master: 950c10dfef394e9f944c6af4280fa7bb (127.20.116.130:42131)
I20260928 14:57:16.154343 6603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:44323
WARNING: no policy specified for kudu/127.20.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
I20260928 14:57:16.177963 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:36501
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:16.261473 6933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:16.261699 6933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:16.261723 6933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:16.262924 6933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:16.262966 6933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:16.262993 6933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:57:16.264406 6933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:36501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:16.264603 6933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:16.264792 6933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:16.269726 6941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:16.269728 6938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:16.269804 6933 server_base.cc:1034] running on GCE node
W20260928 14:57:16.269933 6939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:16.270300 6933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:16.270607 6933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:16.271775 6933 hybrid_clock.cc:648] HybridClock initialized: now 1790607436271762 us; error 31 us; skew 500 ppm
Sep 28 14:57:16 dist-test-slave-pxl6 krb5kdc[6558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607436, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:16.274617 6933 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:57:16.275156 6933 webserver.cc:460] Webserver started at http://127.20.116.131:42737/ using document root <none> and password file <none>
I20260928 14:57:16.275414 6933 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:16.275485 6933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:16.275696 6933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:16.276851 6933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "e685218578fb4a4d91f64b85da08e721"
format_stamp: "Formatted at 2026-09-28 14:57:16 on dist-test-slave-pxl6"
server_key: "12b5cb8d7f04b142bfbe40d900412ef3"
server_key_iv: "eb76b7226c999958b3ac8d432e0418d6"
server_key_version: "encryptionkey@0"
I20260928 14:57:16.277318 6933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "e685218578fb4a4d91f64b85da08e721"
format_stamp: "Formatted at 2026-09-28 14:57:16 on dist-test-slave-pxl6"
server_key: "12b5cb8d7f04b142bfbe40d900412ef3"
server_key_iv: "eb76b7226c999958b3ac8d432e0418d6"
server_key_version: "encryptionkey@0"
I20260928 14:57:16.278808 6933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 14:57:16.279549 6948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:16.279759 6933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:57:16.279863 6933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "e685218578fb4a4d91f64b85da08e721"
format_stamp: "Formatted at 2026-09-28 14:57:16 on dist-test-slave-pxl6"
server_key: "12b5cb8d7f04b142bfbe40d900412ef3"
server_key_iv: "eb76b7226c999958b3ac8d432e0418d6"
server_key_version: "encryptionkey@0"
I20260928 14:57:16.279935 6933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:16.295614 6933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:16.298046 6933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:16.298213 6933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:16.298452 6933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:16.298843 6933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:16.298900 6933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:16.298943 6933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:16.298971 6933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:16.304855 6933 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:35703
I20260928 14:57:16.304924 7061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:35703 every 8 connection(s)
I20260928 14:57:16.305284 6933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:57:16 dist-test-slave-pxl6 krb5kdc[6558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607436, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:57:16.313242 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 6933
I20260928 14:57:16.313371 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:57:16.313553 20946 external_mini_cluster.cc:1428] Setting key 389fe1a7552e9b6895946af32a6b04d9
I20260928 14:57:16.315405 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:16.306034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:43135 (local address 127.20.116.190:36501)
0928 14:57:16.306083 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:16.306085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:16.306275 (+ 190us) server_negotiation.cc:407] Connection header received
0928 14:57:16.306877 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:16.306878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:16.306889 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:16.306944 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:16.307950 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.308661 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.309675 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.309875 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.309937 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:16.312195 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:16.312197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:16.312197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:16.312205 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:16.313957 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:16.314478 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:16.314479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:16.314479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:16.314568 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:16.314825 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:16.314825 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:16.314826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:16.314944 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:16.315000 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:16.315260 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 14:57:16.315338 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:57:16.315651 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:16.306010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:36501 (local address 127.20.116.131:43135)
0928 14:57:16.306166 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:16.306201 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:16.306765 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:16.307000 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:16.307003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:16.307643 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:16.307860 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:16.307865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.308746 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.308747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:16.309581 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:16.309584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.309958 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.309959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:16.310069 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:16.310641 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:16.310651 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:16.312073 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:16.314062 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:16.314072 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:16.314075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:16.314399 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:16.314632 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:16.314633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:16.314634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:16.314760 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:16.315003 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:16.315008 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:16.315142 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 14:57:16.315235 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 14:57:16.315373 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260928 14:57:16.316134 7062 heartbeater.cc:347] Connected to a master server at 127.20.116.190:36501
I20260928 14:57:16.316244 7062 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:16.316435 7062 heartbeater.cc:511] Master 127.20.116.190:36501 requested a full tablet report, sending...
I20260928 14:57:16.316826 6603 ts_manager.cc:194] Registered new tserver with Master: e685218578fb4a4d91f64b85da08e721 (127.20.116.131:35703)
I20260928 14:57:16.317209 6603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:43135
I20260928 14:57:16.325537 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:57:16.333395 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:16.326981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58012 (local address 127.20.116.190:36501)
0928 14:57:16.327082 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:16.327085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:16.327173 (+ 88us) server_negotiation.cc:407] Connection header received
0928 14:57:16.327270 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:16.327284 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:16.327299 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:16.327355 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:16.327923 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.328416 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.329128 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.329333 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.329404 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:16.330922 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:16.330924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:16.330924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:16.330932 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:16.332141 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:16.332540 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:16.332541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:16.332542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:16.332600 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:16.332821 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:16.332822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:16.332822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:16.332972 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:16.333089 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:16.333211 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 14:57:16.333285 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 14:57:16.335067 6603 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:57:16.336879 6603 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:57:16.343787 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:16.339942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42688 (local address 127.20.116.131:35703)
0928 14:57:16.340156 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:16.340161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:16.340172 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:57:16.340211 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:16.340213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:16.340230 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:16.340326 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:16.340483 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.341233 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.342604 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.343324 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.343413 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:16.343479 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:16.343640 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 14:57:16.343686 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":80,"threads_started":1}
I20260928 14:57:16.343791 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:16.339722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:35703 (local address 127.0.0.1:42688)
0928 14:57:16.339947 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:16.339959 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:16.340019 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:16.340331 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:16.340333 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:16.340336 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:16.340390 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:16.340392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.341343 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.341344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:16.342479 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:16.342482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.343409 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.343410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:16.343490 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:16.343506 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 14:57:16.343617 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 14:57:16.343681 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":3,"thread_start_us":81,"threads_started":1}
I20260928 14:57:16.344413 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:16.339501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:41637 (local address 127.0.0.1:49280)
0928 14:57:16.339841 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:16.339851 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:16.339897 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:16.340365 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:16.340366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:16.340368 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:16.340414 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:16.340416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.341924 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.341925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:16.342956 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:16.342959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.344148 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.344149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:16.344216 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:16.344220 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:57:16.344297 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 14:57:16.344340 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":7,"thread_start_us":63,"threads_started":1}
I20260928 14:57:16.344465 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:16.339745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.20.116.129:41637)
0928 14:57:16.340017 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:16.340021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:16.340030 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:57:16.340068 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:16.340070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:16.340083 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:16.340158 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:16.340506 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.341766 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.343065 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.343908 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.344005 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:16.344063 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:16.344328 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 14:57:16.344380 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":100,"threads_started":1}
I20260928 14:57:16.344596 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:16.339501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:42131 (local address 127.0.0.1:35844)
0928 14:57:16.339652 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:16.339700 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:16.339768 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:16.340164 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:16.340168 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:16.340172 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:16.340277 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:16.340280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.341902 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.341903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:16.343007 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:16.343010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.344237 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.344239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:16.344322 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:16.344327 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:57:16.344360 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 14:57:16.344411 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260928 14:57:16.344758 6996 tablet_service.cc:1467] Processing CreateTablet for tablet 540aa005da3a4ea59bc62d4513f2c13e (DEFAULT_TABLE table=test-table [id=8633fb385747435b802c10dac46cd3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:16.345199 6996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 540aa005da3a4ea59bc62d4513f2c13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:16.345184 6728 tablet_service.cc:1467] Processing CreateTablet for tablet 540aa005da3a4ea59bc62d4513f2c13e (DEFAULT_TABLE table=test-table [id=8633fb385747435b802c10dac46cd3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:16.345520 6728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 540aa005da3a4ea59bc62d4513f2c13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:16.345543 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:16.339635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35844 (local address 127.20.116.130:42131)
0928 14:57:16.339952 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:16.339958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:16.339969 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:57:16.340011 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:16.340014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:16.340032 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:16.340142 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:16.340448 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.341773 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.343093 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.344101 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.345297 (+ 1196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:16.345381 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:16.345411 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 14:57:16.345461 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":87,"threads_started":1}
I20260928 14:57:16.347285 6862 tablet_service.cc:1467] Processing CreateTablet for tablet 540aa005da3a4ea59bc62d4513f2c13e (DEFAULT_TABLE table=test-table [id=8633fb385747435b802c10dac46cd3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:16.347625 6862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 540aa005da3a4ea59bc62d4513f2c13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:16.348459 7079 tablet_bootstrap.cc:492] T 540aa005da3a4ea59bc62d4513f2c13e P e685218578fb4a4d91f64b85da08e721: Bootstrap starting.
I20260928 14:57:16.348874 7080 tablet_bootstrap.cc:492] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6: Bootstrap starting.
I20260928 14:57:16.349277 7079 tablet_bootstrap.cc:654] T 540aa005da3a4ea59bc62d4513f2c13e P e685218578fb4a4d91f64b85da08e721: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:16.349608 7080 tablet_bootstrap.cc:654] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:16.349951 7079 log.cc:824] T 540aa005da3a4ea59bc62d4513f2c13e P e685218578fb4a4d91f64b85da08e721: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:16.350241 7080 log.cc:824] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:16.350980 7079 tablet_bootstrap.cc:492] T 540aa005da3a4ea59bc62d4513f2c13e P e685218578fb4a4d91f64b85da08e721: No bootstrap required, opened a new log
I20260928 14:57:16.350905 7083 tablet_bootstrap.cc:492] T 540aa005da3a4ea59bc62d4513f2c13e P 950c10dfef394e9f944c6af4280fa7bb: Bootstrap starting.
I20260928 14:57:16.351058 7079 ts_tablet_manager.cc:1397] T 540aa005da3a4ea59bc62d4513f2c13e P e685218578fb4a4d91f64b85da08e721: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:57:16.351670 7083 tablet_bootstrap.cc:654] T 540aa005da3a4ea59bc62d4513f2c13e P 950c10dfef394e9f944c6af4280fa7bb: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:16.352322 7083 log.cc:824] T 540aa005da3a4ea59bc62d4513f2c13e P 950c10dfef394e9f944c6af4280fa7bb: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:16.353138 7083 tablet_bootstrap.cc:492] T 540aa005da3a4ea59bc62d4513f2c13e P 950c10dfef394e9f944c6af4280fa7bb: No bootstrap required, opened a new log
I20260928 14:57:16.353240 7083 ts_tablet_manager.cc:1397] T 540aa005da3a4ea59bc62d4513f2c13e P 950c10dfef394e9f944c6af4280fa7bb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:57:16.353484 7079 raft_consensus.cc:348] T 540aa005da3a4ea59bc62d4513f2c13e P e685218578fb4a4d91f64b85da08e721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b702ae4de848629ab0258bfb6200c6" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41637 } } peers { permanent_uuid: "e685218578fb4a4d91f64b85da08e721" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 35703 } } peers { permanent_uuid: "950c10dfef394e9f944c6af4280fa7bb" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 42131 } }
I20260928 14:57:16.353688 7079 raft_consensus.cc:374] T 540aa005da3a4ea59bc62d4513f2c13e P e685218578fb4a4d91f64b85da08e721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:16.353747 7079 raft_consensus.cc:729] T 540aa005da3a4ea59bc62d4513f2c13e P e685218578fb4a4d91f64b85da08e721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e685218578fb4a4d91f64b85da08e721, State: Initialized, Role: FOLLOWER
I20260928 14:57:16.353880 7079 consensus_queue.cc:260] T 540aa005da3a4ea59bc62d4513f2c13e P e685218578fb4a4d91f64b85da08e721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b702ae4de848629ab0258bfb6200c6" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41637 } } peers { permanent_uuid: "e685218578fb4a4d91f64b85da08e721" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 35703 } } peers { permanent_uuid: "950c10dfef394e9f944c6af4280fa7bb" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 42131 } }
I20260928 14:57:16.354187 7062 heartbeater.cc:503] Master 127.20.116.190:36501 was elected leader, sending a full tablet report...
I20260928 14:57:16.354187 7079 ts_tablet_manager.cc:1428] T 540aa005da3a4ea59bc62d4513f2c13e P e685218578fb4a4d91f64b85da08e721: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:57:16.354820 7080 tablet_bootstrap.cc:492] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6: No bootstrap required, opened a new log
I20260928 14:57:16.354902 7080 ts_tablet_manager.cc:1397] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 14:57:16.355669 7083 raft_consensus.cc:348] T 540aa005da3a4ea59bc62d4513f2c13e P 950c10dfef394e9f944c6af4280fa7bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b702ae4de848629ab0258bfb6200c6" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41637 } } peers { permanent_uuid: "e685218578fb4a4d91f64b85da08e721" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 35703 } } peers { permanent_uuid: "950c10dfef394e9f944c6af4280fa7bb" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 42131 } }
I20260928 14:57:16.355810 7083 raft_consensus.cc:374] T 540aa005da3a4ea59bc62d4513f2c13e P 950c10dfef394e9f944c6af4280fa7bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:16.355834 7083 raft_consensus.cc:729] T 540aa005da3a4ea59bc62d4513f2c13e P 950c10dfef394e9f944c6af4280fa7bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 950c10dfef394e9f944c6af4280fa7bb, State: Initialized, Role: FOLLOWER
I20260928 14:57:16.355973 7083 consensus_queue.cc:260] T 540aa005da3a4ea59bc62d4513f2c13e P 950c10dfef394e9f944c6af4280fa7bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b702ae4de848629ab0258bfb6200c6" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41637 } } peers { permanent_uuid: "e685218578fb4a4d91f64b85da08e721" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 35703 } } peers { permanent_uuid: "950c10dfef394e9f944c6af4280fa7bb" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 42131 } }
I20260928 14:57:16.356271 7083 ts_tablet_manager.cc:1428] T 540aa005da3a4ea59bc62d4513f2c13e P 950c10dfef394e9f944c6af4280fa7bb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:57:16.356287 6928 heartbeater.cc:503] Master 127.20.116.190:36501 was elected leader, sending a full tablet report...
I20260928 14:57:16.356956 7080 raft_consensus.cc:348] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b702ae4de848629ab0258bfb6200c6" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41637 } } peers { permanent_uuid: "e685218578fb4a4d91f64b85da08e721" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 35703 } } peers { permanent_uuid: "950c10dfef394e9f944c6af4280fa7bb" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 42131 } }
I20260928 14:57:16.357126 7080 raft_consensus.cc:374] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:16.357160 7080 raft_consensus.cc:729] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2b702ae4de848629ab0258bfb6200c6, State: Initialized, Role: FOLLOWER
I20260928 14:57:16.357273 7080 consensus_queue.cc:260] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b702ae4de848629ab0258bfb6200c6" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41637 } } peers { permanent_uuid: "e685218578fb4a4d91f64b85da08e721" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 35703 } } peers { permanent_uuid: "950c10dfef394e9f944c6af4280fa7bb" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 42131 } }
I20260928 14:57:16.357573 7080 ts_tablet_manager.cc:1428] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 14:57:16.357607 6794 heartbeater.cc:503] Master 127.20.116.190:36501 was elected leader, sending a full tablet report...
W20260928 14:57:16.392596 6929 tablet.cc:2369] T 540aa005da3a4ea59bc62d4513f2c13e P 950c10dfef394e9f944c6af4280fa7bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:57:16.471297 6795 tablet.cc:2369] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:57:16.536015 7087 raft_consensus.cc:482] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:57:16.536149 7087 raft_consensus.cc:504] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b702ae4de848629ab0258bfb6200c6" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41637 } } peers { permanent_uuid: "e685218578fb4a4d91f64b85da08e721" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 35703 } } peers { permanent_uuid: "950c10dfef394e9f944c6af4280fa7bb" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 42131 } }
I20260928 14:57:16.536583 7087 leader_election.cc:288] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e685218578fb4a4d91f64b85da08e721 (127.20.116.131:35703), 950c10dfef394e9f944c6af4280fa7bb (127.20.116.130:42131)
I20260928 14:57:16.539566 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:16.536865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:44947 (local address 127.20.116.131:35703)
0928 14:57:16.536930 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:16.536932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:16.537014 (+ 82us) server_negotiation.cc:407] Connection header received
0928 14:57:16.537099 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:16.537100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:16.537110 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:16.537173 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:16.537343 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.537825 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.538846 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.539220 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.539302 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:16.539313 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:16.539453 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 14:57:16.539491 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:57:16.539587 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:16.536911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:42021 (local address 127.20.116.130:42131)
0928 14:57:16.536960 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:16.536962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:16.537035 (+ 73us) server_negotiation.cc:407] Connection header received
0928 14:57:16.537108 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:16.537110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:16.537122 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:16.537185 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:16.537360 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.537817 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.538854 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.539232 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.539328 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:16.539337 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:16.539478 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 14:57:16.539518 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 14:57:16.539618 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:16.536810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:42131 (local address 127.20.116.129:42021)
0928 14:57:16.536963 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:16.536972 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:16.537051 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:16.537204 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:16.537205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:16.537207 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:16.537292 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:16.537294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.537898 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.537899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:16.538763 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:16.538765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.539332 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.539333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:16.539398 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:16.539401 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:57:16.539470 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 14:57:16.539513 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":7,"thread_start_us":79,"threads_started":1}
I20260928 14:57:16.539631 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:16.536760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:35703 (local address 127.20.116.129:44947)
0928 14:57:16.536933 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:16.536943 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:16.537030 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:16.537179 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:16.537180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:16.537182 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:16.537275 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:16.537277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.537907 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.537908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:16.538762 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:16.538764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.539310 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.539311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:16.539383 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:16.539386 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:57:16.539447 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 14:57:16.539489 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260928 14:57:16.539984 7016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "540aa005da3a4ea59bc62d4513f2c13e" candidate_uuid: "e2b702ae4de848629ab0258bfb6200c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e685218578fb4a4d91f64b85da08e721" is_pre_election: true
I20260928 14:57:16.539986 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "540aa005da3a4ea59bc62d4513f2c13e" candidate_uuid: "e2b702ae4de848629ab0258bfb6200c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "950c10dfef394e9f944c6af4280fa7bb" is_pre_election: true
I20260928 14:57:16.540151 7016 raft_consensus.cc:2454] T 540aa005da3a4ea59bc62d4513f2c13e P e685218578fb4a4d91f64b85da08e721 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2b702ae4de848629ab0258bfb6200c6 in term 0.
I20260928 14:57:16.540151 6882 raft_consensus.cc:2454] T 540aa005da3a4ea59bc62d4513f2c13e P 950c10dfef394e9f944c6af4280fa7bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2b702ae4de848629ab0258bfb6200c6 in term 0.
I20260928 14:57:16.540400 6682 leader_election.cc:302] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2b702ae4de848629ab0258bfb6200c6, e685218578fb4a4d91f64b85da08e721; no voters:
I20260928 14:57:16.540565 7087 raft_consensus.cc:2781] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:57:16.540627 7087 raft_consensus.cc:482] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:57:16.540670 7087 raft_consensus.cc:3037] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:16.541402 7087 raft_consensus.cc:504] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b702ae4de848629ab0258bfb6200c6" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41637 } } peers { permanent_uuid: "e685218578fb4a4d91f64b85da08e721" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 35703 } } peers { permanent_uuid: "950c10dfef394e9f944c6af4280fa7bb" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 42131 } }
I20260928 14:57:16.541548 7087 leader_election.cc:288] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6 [CANDIDATE]: Term 1 election: Requested vote from peers e685218578fb4a4d91f64b85da08e721 (127.20.116.131:35703), 950c10dfef394e9f944c6af4280fa7bb (127.20.116.130:42131)
I20260928 14:57:16.541720 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "540aa005da3a4ea59bc62d4513f2c13e" candidate_uuid: "e2b702ae4de848629ab0258bfb6200c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "950c10dfef394e9f944c6af4280fa7bb"
I20260928 14:57:16.541714 7016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "540aa005da3a4ea59bc62d4513f2c13e" candidate_uuid: "e2b702ae4de848629ab0258bfb6200c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e685218578fb4a4d91f64b85da08e721"
I20260928 14:57:16.541805 6882 raft_consensus.cc:3037] T 540aa005da3a4ea59bc62d4513f2c13e P 950c10dfef394e9f944c6af4280fa7bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:16.541805 7016 raft_consensus.cc:3037] T 540aa005da3a4ea59bc62d4513f2c13e P e685218578fb4a4d91f64b85da08e721 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:16.542748 7016 raft_consensus.cc:2454] T 540aa005da3a4ea59bc62d4513f2c13e P e685218578fb4a4d91f64b85da08e721 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2b702ae4de848629ab0258bfb6200c6 in term 1.
I20260928 14:57:16.542748 6882 raft_consensus.cc:2454] T 540aa005da3a4ea59bc62d4513f2c13e P 950c10dfef394e9f944c6af4280fa7bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2b702ae4de848629ab0258bfb6200c6 in term 1.
I20260928 14:57:16.542924 6684 leader_election.cc:302] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 950c10dfef394e9f944c6af4280fa7bb, e2b702ae4de848629ab0258bfb6200c6; no voters:
I20260928 14:57:16.543061 7087 raft_consensus.cc:2781] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:16.543249 7087 raft_consensus.cc:686] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6 [term 1 LEADER]: Becoming Leader. State: Replica: e2b702ae4de848629ab0258bfb6200c6, State: Running, Role: LEADER
I20260928 14:57:16.543349 7087 consensus_queue.cc:237] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b702ae4de848629ab0258bfb6200c6" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41637 } } peers { permanent_uuid: "e685218578fb4a4d91f64b85da08e721" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 35703 } } peers { permanent_uuid: "950c10dfef394e9f944c6af4280fa7bb" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 42131 } }
I20260928 14:57:16.544271 6602 catalog_manager.cc:5495] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2b702ae4de848629ab0258bfb6200c6 (127.20.116.129). New cstate: current_term: 1 leader_uuid: "e2b702ae4de848629ab0258bfb6200c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b702ae4de848629ab0258bfb6200c6" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e685218578fb4a4d91f64b85da08e721" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 35703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "950c10dfef394e9f944c6af4280fa7bb" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 42131 } health_report { overall_health: UNKNOWN } } }
W20260928 14:57:16.555855 7063 tablet.cc:2369] T 540aa005da3a4ea59bc62d4513f2c13e P e685218578fb4a4d91f64b85da08e721: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:57:16.573112 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:16.570839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.20.116.129:41637)
0928 14:57:16.570909 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:16.570911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:16.570922 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:57:16.570971 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:16.570971 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:16.570999 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:16.571068 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:16.571172 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.571640 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.572352 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.572538 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.572589 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:16.572703 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:16.572796 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:16.572871 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:16.572983 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 14:57:16.573011 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:57:16.575310 6882 raft_consensus.cc:1261] T 540aa005da3a4ea59bc62d4513f2c13e P 950c10dfef394e9f944c6af4280fa7bb [term 1 FOLLOWER]: Refusing update from remote peer e2b702ae4de848629ab0258bfb6200c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:16.575311 7016 raft_consensus.cc:1261] T 540aa005da3a4ea59bc62d4513f2c13e P e685218578fb4a4d91f64b85da08e721 [term 1 FOLLOWER]: Refusing update from remote peer e2b702ae4de848629ab0258bfb6200c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:16.575711 7087 consensus_queue.cc:1035] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "950c10dfef394e9f944c6af4280fa7bb" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 42131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:16.575879 7090 consensus_queue.cc:1035] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e685218578fb4a4d91f64b85da08e721" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 35703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:16.578903 7093 mvcc.cc:205] Tried to move back new op lower bound from 7334328060208848896 to 7334328060082237440. Current Snapshot: MvccSnapshot[applied={T|T < 7334328060208848896}]
I20260928 14:57:16.578352 7095 mvcc.cc:205] Tried to move back new op lower bound from 7334328060208848896 to 7334328060082237440. Current Snapshot: MvccSnapshot[applied={T|T < 7334328060208848896}]
I20260928 14:57:16.581701 6602 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:16.581806 6602 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:16.582657 6602 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 759dcb9340f34cc4947af6e53a8b95ae: Sending DeleteTablet for 3 replicas of tablet 540aa005da3a4ea59bc62d4513f2c13e
I20260928 14:57:16.583086 6862 tablet_service.cc:1514] Processing DeleteTablet for tablet 540aa005da3a4ea59bc62d4513f2c13e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:16 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:35844
I20260928 14:57:16.583114 6728 tablet_service.cc:1514] Processing DeleteTablet for tablet 540aa005da3a4ea59bc62d4513f2c13e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:16 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:49280
I20260928 14:57:16.583344 7103 tablet_replica.cc:330] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6: stopping tablet replica
I20260928 14:57:16.583446 7103 raft_consensus.cc:2229] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:57:16.583415 6996 tablet_service.cc:1514] Processing DeleteTablet for tablet 540aa005da3a4ea59bc62d4513f2c13e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:16 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:42688
I20260928 14:57:16.583621 7103 raft_consensus.cc:2258] T 540aa005da3a4ea59bc62d4513f2c13e P e2b702ae4de848629ab0258bfb6200c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:16.583786 7104 tablet_replica.cc:330] T 540aa005da3a4ea59bc62d4513f2c13e P 950c10dfef394e9f944c6af4280fa7bb: stopping tablet replica
I20260928 14:57:16.583882 7104 raft_consensus.cc:2229] T 540aa005da3a4ea59bc62d4513f2c13e P 950c10dfef394e9f944c6af4280fa7bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:16.584002 7104 raft_consensus.cc:2258] T 540aa005da3a4ea59bc62d4513f2c13e P 950c10dfef394e9f944c6af4280fa7bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:16.584043 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 6665
I20260928 14:57:16.585583 7105 tablet_replica.cc:330] T 540aa005da3a4ea59bc62d4513f2c13e P e685218578fb4a4d91f64b85da08e721: stopping tablet replica
I20260928 14:57:16.585652 7104 ts_tablet_manager.cc:1905] T 540aa005da3a4ea59bc62d4513f2c13e P 950c10dfef394e9f944c6af4280fa7bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:16.585734 7105 raft_consensus.cc:2229] T 540aa005da3a4ea59bc62d4513f2c13e P e685218578fb4a4d91f64b85da08e721 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:16.585842 7105 raft_consensus.cc:2258] T 540aa005da3a4ea59bc62d4513f2c13e P e685218578fb4a4d91f64b85da08e721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:16.586308 7105 ts_tablet_manager.cc:1905] T 540aa005da3a4ea59bc62d4513f2c13e P e685218578fb4a4d91f64b85da08e721: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:16.587723 7105 ts_tablet_manager.cc:1918] T 540aa005da3a4ea59bc62d4513f2c13e P e685218578fb4a4d91f64b85da08e721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:16.587745 7104 ts_tablet_manager.cc:1918] T 540aa005da3a4ea59bc62d4513f2c13e P 950c10dfef394e9f944c6af4280fa7bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:16.587816 7104 log.cc:1196] T 540aa005da3a4ea59bc62d4513f2c13e P 950c10dfef394e9f944c6af4280fa7bb: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/540aa005da3a4ea59bc62d4513f2c13e
I20260928 14:57:16.587816 7105 log.cc:1196] T 540aa005da3a4ea59bc62d4513f2c13e P e685218578fb4a4d91f64b85da08e721: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/540aa005da3a4ea59bc62d4513f2c13e
I20260928 14:57:16.588104 7105 ts_tablet_manager.cc:1939] T 540aa005da3a4ea59bc62d4513f2c13e P e685218578fb4a4d91f64b85da08e721: Deleting consensus metadata
I20260928 14:57:16.588119 7104 ts_tablet_manager.cc:1939] T 540aa005da3a4ea59bc62d4513f2c13e P 950c10dfef394e9f944c6af4280fa7bb: Deleting consensus metadata
I20260928 14:57:16.588567 6590 catalog_manager.cc:4841] TS 950c10dfef394e9f944c6af4280fa7bb (127.20.116.130:42131): tablet 540aa005da3a4ea59bc62d4513f2c13e (table test-table [id=8633fb385747435b802c10dac46cd3c0]) successfully deleted
I20260928 14:57:16.588716 6588 catalog_manager.cc:4841] TS e685218578fb4a4d91f64b85da08e721 (127.20.116.131:35703): tablet 540aa005da3a4ea59bc62d4513f2c13e (table test-table [id=8633fb385747435b802c10dac46cd3c0]) successfully deleted
W20260928 14:57:16.590938 6588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.116.129:41637 (error 108)
I20260928 14:57:16.591522 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 6799
I20260928 14:57:16.591729 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:16.591542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:41637 (local address 127.0.0.1:49292)
0928 14:57:16.591645 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:16.591688 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.129:41637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 14:57:16.592163 6588 catalog_manager.cc:4584] TS e2b702ae4de848629ab0258bfb6200c6 (127.20.116.129:41637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 540aa005da3a4ea59bc62d4513f2c13e: Network error: Client connection negotiation failed: client connection to 127.20.116.129:41637: connect: Connection refused (error 111)
I20260928 14:57:16.597172 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 6933
I20260928 14:57:16.602053 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 6571
2026-09-28T14:57:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2147 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T14:57:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:16Z Disabled control of system clock
I20260928 14:57:16.630146 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:43921
--webserver_interface=127.20.116.190
--webserver_port=0
--builtin_ntp_servers=127.20.116.148:36305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:43921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:57:16.712579 7110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:16.712826 7110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:16.712847 7110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:16.713981 7110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:16.714015 7110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:16.714028 7110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:16.714041 7110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:16.714053 7110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:16.715382 7110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:36305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:43921
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:43921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:16.715591 7110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:16.715848 7110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:16.720736 7116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:16.720894 7118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:16.720907 7115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:16.721087 7110 server_base.cc:1034] running on GCE node
I20260928 14:57:16.721333 7110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:16.721632 7110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:16.722791 7110 hybrid_clock.cc:648] HybridClock initialized: now 1790607436722777 us; error 43 us; skew 500 ppm
I20260928 14:57:16.724159 7110 webserver.cc:460] Webserver started at http://127.20.116.190:40785/ using document root <none> and password file <none>
I20260928 14:57:16.724421 7110 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:16.724488 7110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:16.724650 7110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:16.725826 7110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "0338b2a995e24310a102c0dd3267669d"
format_stamp: "Formatted at 2026-09-28 14:57:16 on dist-test-slave-pxl6"
server_key: "bd769d58e6a500c4ce005422746b1b0d"
server_key_iv: "152956c2b4db4365f33fd77929dd3304"
server_key_version: "encryptionkey@0"
I20260928 14:57:16.726190 7110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "0338b2a995e24310a102c0dd3267669d"
format_stamp: "Formatted at 2026-09-28 14:57:16 on dist-test-slave-pxl6"
server_key: "bd769d58e6a500c4ce005422746b1b0d"
server_key_iv: "152956c2b4db4365f33fd77929dd3304"
server_key_version: "encryptionkey@0"
I20260928 14:57:16.727546 7110 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:57:16.728315 7124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:16.728478 7110 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:57:16.728579 7110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "0338b2a995e24310a102c0dd3267669d"
format_stamp: "Formatted at 2026-09-28 14:57:16 on dist-test-slave-pxl6"
server_key: "bd769d58e6a500c4ce005422746b1b0d"
server_key_iv: "152956c2b4db4365f33fd77929dd3304"
server_key_version: "encryptionkey@0"
I20260928 14:57:16.728650 7110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:16.737908 7110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:16.738240 7110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:16.738389 7110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:16.742317 7110 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:43921
I20260928 14:57:16.742355 7176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:43921 every 8 connection(s)
I20260928 14:57:16.742777 7110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:16.743767 7177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:16.744392 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 7110
I20260928 14:57:16.744525 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:16.744760 20946 external_mini_cluster.cc:1428] Setting key 975cb772cc8f2aeee42a7e085e413127
I20260928 14:57:16.746901 7177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d: Bootstrap starting.
I20260928 14:57:16.747581 7177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:16.747913 7177 log.cc:824] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:16.748665 7177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d: No bootstrap required, opened a new log
I20260928 14:57:16.750473 7180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:16.745344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51562 (local address 127.20.116.190:43921)
0928 14:57:16.745542 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:16.745549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:16.745576 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:57:16.746004 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:16.746012 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:16.746073 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:16.746200 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:16.746421 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.747375 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.748531 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.748988 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.749080 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:16.749805 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:16.749809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:16.749809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:16.749972 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:16.750006 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:16.750017 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:16.750069 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:16.750169 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 14:57:16.750289 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260928 14:57:16.751118 7177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0338b2a995e24310a102c0dd3267669d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43921 } }
I20260928 14:57:16.751289 7177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:16.751335 7177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0338b2a995e24310a102c0dd3267669d, State: Initialized, Role: FOLLOWER
I20260928 14:57:16.751453 7177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0338b2a995e24310a102c0dd3267669d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43921 } }
I20260928 14:57:16.751528 7177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:16.751570 7177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:16.751618 7177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:16.752189 7177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0338b2a995e24310a102c0dd3267669d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43921 } }
I20260928 14:57:16.752300 7177 leader_election.cc:302] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0338b2a995e24310a102c0dd3267669d; no voters:
I20260928 14:57:16.752487 7177 leader_election.cc:288] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:16.752559 7182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:16.752758 7182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d [term 1 LEADER]: Becoming Leader. State: Replica: 0338b2a995e24310a102c0dd3267669d, State: Running, Role: LEADER
I20260928 14:57:16.752767 7177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:16.752882 7182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0338b2a995e24310a102c0dd3267669d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43921 } }
I20260928 14:57:16.753269 7183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0338b2a995e24310a102c0dd3267669d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0338b2a995e24310a102c0dd3267669d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43921 } } }
I20260928 14:57:16.753363 7183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:16.753525 7184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0338b2a995e24310a102c0dd3267669d. Latest consensus state: current_term: 1 leader_uuid: "0338b2a995e24310a102c0dd3267669d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0338b2a995e24310a102c0dd3267669d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 43921 } } }
I20260928 14:57:16.753597 7184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:16.753867 7191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:16.754315 7191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:16.756103 7191 catalog_manager.cc:1347] Generated new cluster ID: 9261a2f75f2e4c3e9191a9f83e53db16
I20260928 14:57:16.756186 7191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:16.770300 7191 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:16.770946 7191 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:16.773943 7191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d: Generated new TSK 0
I20260928 14:57:16.774205 7191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:16.776700 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:43921
--builtin_ntp_servers=127.20.116.148:36305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:57:16.861624 7201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:16.861853 7201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:16.861897 7201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:16.863247 7201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:16.863294 7201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:16.863323 7201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:57:16.864768 7201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:36305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:43921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:16.865038 7201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:16.865373 7201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:16.870622 7207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:16.870622 7206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:16.870640 7209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:16.870939 7201 server_base.cc:1034] running on GCE node
I20260928 14:57:16.871239 7201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:16.871522 7201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:16.872736 7201 hybrid_clock.cc:648] HybridClock initialized: now 1790607436872721 us; error 33 us; skew 500 ppm
I20260928 14:57:16.874080 7201 webserver.cc:460] Webserver started at http://127.20.116.129:46639/ using document root <none> and password file <none>
I20260928 14:57:16.874367 7201 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:16.874442 7201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:16.874645 7201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:16.876206 7201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "73e432ec15a046e58097af104548da98"
format_stamp: "Formatted at 2026-09-28 14:57:16 on dist-test-slave-pxl6"
server_key: "962a11d287530d01128947951d1aca97"
server_key_iv: "6e734a441bd88066ba041f547498bdcf"
server_key_version: "encryptionkey@0"
I20260928 14:57:16.876633 7201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "73e432ec15a046e58097af104548da98"
format_stamp: "Formatted at 2026-09-28 14:57:16 on dist-test-slave-pxl6"
server_key: "962a11d287530d01128947951d1aca97"
server_key_iv: "6e734a441bd88066ba041f547498bdcf"
server_key_version: "encryptionkey@0"
I20260928 14:57:16.878093 7201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:16.878958 7215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:16.879154 7201 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:57:16.879264 7201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "73e432ec15a046e58097af104548da98"
format_stamp: "Formatted at 2026-09-28 14:57:16 on dist-test-slave-pxl6"
server_key: "962a11d287530d01128947951d1aca97"
server_key_iv: "6e734a441bd88066ba041f547498bdcf"
server_key_version: "encryptionkey@0"
I20260928 14:57:16.879340 7201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:16.901985 7201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:16.902359 7201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:16.902513 7201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:16.902820 7201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:16.903179 7201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:16.903230 7201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:16.903272 7201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:16.903302 7201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:16.909308 7201 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:38143
I20260928 14:57:16.909355 7328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:38143 every 8 connection(s)
I20260928 14:57:16.909716 7201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:57:16.912019 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 7201
I20260928 14:57:16.912158 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:57:16.912385 20946 external_mini_cluster.cc:1428] Setting key bc003bf8ad79272b38a36dbf3730e0bd
I20260928 14:57:16.913491 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:43921
--builtin_ntp_servers=127.20.116.148:36305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:57:16.915741 7180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:16.910559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:42127 (local address 127.20.116.190:43921)
0928 14:57:16.910610 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:16.910612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:16.910895 (+ 283us) server_negotiation.cc:407] Connection header received
0928 14:57:16.911477 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:16.911479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:16.911498 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:16.911550 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:16.912369 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.913029 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.914051 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.914260 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.914332 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:16.915402 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:16.915404 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:16.915404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:16.915410 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:16.915428 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:16.915432 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:16.915488 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:16.915641 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 14:57:16.915673 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:57:16.915968 7331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:16.910525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43921 (local address 127.20.116.129:42127)
0928 14:57:16.910769 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:16.910807 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:16.911363 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:16.911627 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:16.911630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:16.911638 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:16.912269 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:16.912276 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.913129 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.913130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:16.913958 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:16.913960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:16.914357 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:16.914358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:16.914502 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:16.915223 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:57:16.915224 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:16.915270 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:57:16.915303 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:16.915499 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:16.915506 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:16.915509 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:57:16.915580 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 14:57:16.915707 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":102,"threads_started":1}
I20260928 14:57:16.916572 7329 heartbeater.cc:347] Connected to a master server at 127.20.116.190:43921
I20260928 14:57:16.916692 7329 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:16.916904 7329 heartbeater.cc:511] Master 127.20.116.190:43921 requested a full tablet report, sending...
I20260928 14:57:16.917519 7141 ts_manager.cc:194] Registered new tserver with Master: 73e432ec15a046e58097af104548da98 (127.20.116.129:38143)
I20260928 14:57:16.918229 7141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.116.129:42127
W20260928 14:57:17.000151 7332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:17.000442 7332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:17.000480 7332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:17.001745 7332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:17.001813 7332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:17.001857 7332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:57:17.003371 7332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:36305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:43921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:17.003659 7332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:17.003988 7332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:17.009253 7338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:17.009217 7337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:17.009300 7332 server_base.cc:1034] running on GCE node
W20260928 14:57:17.009205 7340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:17.009596 7332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:17.009877 7332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:17.011049 7332 hybrid_clock.cc:648] HybridClock initialized: now 1790607437011030 us; error 30 us; skew 500 ppm
I20260928 14:57:17.012454 7332 webserver.cc:460] Webserver started at http://127.20.116.130:33693/ using document root <none> and password file <none>
I20260928 14:57:17.012712 7332 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:17.012792 7332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:17.012936 7332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:17.014061 7332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "d4e83b2c8e48473f8ad096af9fcc5fec"
format_stamp: "Formatted at 2026-09-28 14:57:17 on dist-test-slave-pxl6"
server_key: "47046ebfb0291ad8be106c2c861b54bf"
server_key_iv: "bd59808399b45fd54a8d5cf88a235018"
server_key_version: "encryptionkey@0"
I20260928 14:57:17.014464 7332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "d4e83b2c8e48473f8ad096af9fcc5fec"
format_stamp: "Formatted at 2026-09-28 14:57:17 on dist-test-slave-pxl6"
server_key: "47046ebfb0291ad8be106c2c861b54bf"
server_key_iv: "bd59808399b45fd54a8d5cf88a235018"
server_key_version: "encryptionkey@0"
I20260928 14:57:17.015872 7332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:57:17.016635 7346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:17.016834 7332 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:57:17.016944 7332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "d4e83b2c8e48473f8ad096af9fcc5fec"
format_stamp: "Formatted at 2026-09-28 14:57:17 on dist-test-slave-pxl6"
server_key: "47046ebfb0291ad8be106c2c861b54bf"
server_key_iv: "bd59808399b45fd54a8d5cf88a235018"
server_key_version: "encryptionkey@0"
I20260928 14:57:17.017021 7332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:17.028410 7332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:17.028756 7332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:17.028914 7332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:17.029160 7332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:17.029487 7332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:17.029536 7332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:17.029572 7332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:17.029613 7332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:17.035607 7332 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:33519
I20260928 14:57:17.035655 7459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:33519 every 8 connection(s)
I20260928 14:57:17.036024 7332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:57:17.039201 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 7332
I20260928 14:57:17.039315 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:57:17.039516 20946 external_mini_cluster.cc:1428] Setting key 6d2e44959a0330f2943a4606ac317e95
I20260928 14:57:17.040669 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:43921
--builtin_ntp_servers=127.20.116.148:36305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:57:17.041735 7180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:17.036788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:38565 (local address 127.20.116.190:43921)
0928 14:57:17.036870 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:17.036873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:17.037066 (+ 193us) server_negotiation.cc:407] Connection header received
0928 14:57:17.037557 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:17.037559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:17.037569 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:17.037625 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:17.038263 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.038840 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.040069 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.040388 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.040446 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:17.041481 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:17.041483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:17.041483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:17.041488 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:17.041503 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:17.041507 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:17.041548 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:17.041648 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 14:57:17.041674 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:57:17.042074 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:17.036749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43921 (local address 127.20.116.130:38565)
0928 14:57:17.036948 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:17.036984 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:17.037474 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:17.037679 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:17.037683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:17.037691 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:17.038191 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:17.038195 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.038950 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.038952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:17.039989 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:17.039992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.040508 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.040510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:17.040697 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:17.041304 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:57:17.041305 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:17.041346 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:57:17.041376 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:17.041564 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:17.041570 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:17.041573 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:57:17.041647 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 14:57:17.041783 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
I20260928 14:57:17.042618 7460 heartbeater.cc:347] Connected to a master server at 127.20.116.190:43921
I20260928 14:57:17.042735 7460 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:17.042968 7460 heartbeater.cc:511] Master 127.20.116.190:43921 requested a full tablet report, sending...
I20260928 14:57:17.043371 7141 ts_manager.cc:194] Registered new tserver with Master: d4e83b2c8e48473f8ad096af9fcc5fec (127.20.116.130:33519)
I20260928 14:57:17.043784 7141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.116.130:38565
W20260928 14:57:17.125578 7463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:17.125836 7463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:17.125876 7463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:17.127163 7463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:17.127233 7463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:17.127277 7463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:57:17.128746 7463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:36305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:43921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:17.129035 7463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:17.129352 7463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:17.134886 7468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:17.134886 7471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:17.134867 7469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:17.134905 7463 server_base.cc:1034] running on GCE node
I20260928 14:57:17.135277 7463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:17.135530 7463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:17.136674 7463 hybrid_clock.cc:648] HybridClock initialized: now 1790607437136656 us; error 33 us; skew 500 ppm
I20260928 14:57:17.137997 7463 webserver.cc:460] Webserver started at http://127.20.116.131:44991/ using document root <none> and password file <none>
I20260928 14:57:17.138247 7463 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:17.138296 7463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:17.138454 7463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:17.139632 7463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "3f39d1907d8545e094d0e28b036edd82"
format_stamp: "Formatted at 2026-09-28 14:57:17 on dist-test-slave-pxl6"
server_key: "b074e9f50d60ee4629937e2fd271fe3a"
server_key_iv: "37c413d7fd22282d155e1094654412b5"
server_key_version: "encryptionkey@0"
I20260928 14:57:17.140005 7463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "3f39d1907d8545e094d0e28b036edd82"
format_stamp: "Formatted at 2026-09-28 14:57:17 on dist-test-slave-pxl6"
server_key: "b074e9f50d60ee4629937e2fd271fe3a"
server_key_iv: "37c413d7fd22282d155e1094654412b5"
server_key_version: "encryptionkey@0"
I20260928 14:57:17.141433 7463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:17.142266 7477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:17.142458 7463 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:57:17.142575 7463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "3f39d1907d8545e094d0e28b036edd82"
format_stamp: "Formatted at 2026-09-28 14:57:17 on dist-test-slave-pxl6"
server_key: "b074e9f50d60ee4629937e2fd271fe3a"
server_key_iv: "37c413d7fd22282d155e1094654412b5"
server_key_version: "encryptionkey@0"
I20260928 14:57:17.142665 7463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:17.153326 7463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:17.153635 7463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:17.153743 7463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:17.153986 7463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:17.154340 7463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:17.154374 7463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:17.154397 7463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:17.154409 7463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:17.160549 7463 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:37177
I20260928 14:57:17.160647 7590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:37177 every 8 connection(s)
I20260928 14:57:17.160980 7463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:57:17.166649 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 7463
I20260928 14:57:17.166767 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:57:17.166863 7180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:17.161926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:39429 (local address 127.20.116.190:43921)
0928 14:57:17.162026 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:17.162029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:17.162264 (+ 235us) server_negotiation.cc:407] Connection header received
0928 14:57:17.162979 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:17.162981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:17.162994 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:17.163050 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:17.163797 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.164308 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.165340 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.165540 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.165615 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:17.166514 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:17.166516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:17.166516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:17.166522 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:17.166562 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:17.166566 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:17.166622 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:17.166715 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 14:57:17.166746 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:57:17.166997 20946 external_mini_cluster.cc:1428] Setting key 9a5ec3df274ac46c03b95405f85bd410
I20260928 14:57:17.167099 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:17.161877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:43921 (local address 127.20.116.131:39429)
0928 14:57:17.162131 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:17.162170 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:17.162864 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:17.163131 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:17.163135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:17.163144 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:17.163691 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:17.163696 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.164410 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.164411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:17.165224 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:17.165227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.165626 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.165628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:17.165782 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:17.166370 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:57:17.166370 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:17.166399 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:57:17.166423 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:17.166625 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:17.166630 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:17.166634 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:57:17.166719 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 14:57:17.166832 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":101,"threads_started":1}
I20260928 14:57:17.167594 7591 heartbeater.cc:347] Connected to a master server at 127.20.116.190:43921
I20260928 14:57:17.167709 7591 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:17.167929 7591 heartbeater.cc:511] Master 127.20.116.190:43921 requested a full tablet report, sending...
I20260928 14:57:17.168335 7140 ts_manager.cc:194] Registered new tserver with Master: 3f39d1907d8545e094d0e28b036edd82 (127.20.116.131:37177)
I20260928 14:57:17.168722 7140 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.116.131:39429
I20260928 14:57:17.179226 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:57:17.183986 7180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:17.180712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51568 (local address 127.20.116.190:43921)
0928 14:57:17.180793 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:17.180796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:17.180862 (+ 66us) server_negotiation.cc:407] Connection header received
0928 14:57:17.180949 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:17.180950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:17.180961 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:17.181030 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:17.181217 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.181764 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.182665 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.182881 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.182958 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:17.183663 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:17.183664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:17.183664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:17.183670 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:17.183684 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:17.183688 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:17.183811 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:17.183876 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 14:57:17.183888 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:57:17.185870 7140 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:57:17.187578 7140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:57:17.194226 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:17.189991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39426 (local address 127.20.116.130:33519)
0928 14:57:17.190429 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:17.190432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:17.190502 (+ 70us) server_negotiation.cc:407] Connection header received
0928 14:57:17.190588 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:17.190591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:17.190607 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:17.190683 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:17.190891 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.192055 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.193327 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.193817 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.193910 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:17.194007 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:17.194092 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 14:57:17.194133 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":121,"threads_started":1}
I20260928 14:57:17.194219 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:17.189874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:33519 (local address 127.0.0.1:39426)
0928 14:57:17.190422 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:17.190433 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:17.190496 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:17.190700 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:17.190704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:17.190706 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:17.190786 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:17.190788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.192140 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.192141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:17.193235 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:17.193237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.193908 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.193909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:17.193985 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:17.193998 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 14:57:17.194075 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 14:57:17.194131 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":29,"thread_start_us":59,"threads_started":1}
I20260928 14:57:17.194382 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:17.190206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47128 (local address 127.20.116.131:37177)
0928 14:57:17.190376 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:17.190382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:17.190663 (+ 281us) server_negotiation.cc:407] Connection header received
0928 14:57:17.190705 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:17.190707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:17.190721 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:17.190810 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:17.190936 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.192065 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.193138 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.193872 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.193946 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:17.194037 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:17.194260 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 14:57:17.194296 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260928 14:57:17.194523 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:17.190231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35356 (local address 127.20.116.129:38143)
0928 14:57:17.190414 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:17.190419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:17.190431 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:57:17.190477 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:17.190479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:17.190503 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:17.190982 (+ 479us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:17.191085 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.191776 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.193279 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.193979 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.194285 (+ 306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:17.194377 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:17.194402 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 14:57:17.194445 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260928 14:57:17.194631 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:17.189874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:38143 (local address 127.0.0.1:35356)
0928 14:57:17.190047 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:17.190119 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:17.190191 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:17.190972 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:17.190972 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:17.190974 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:17.191011 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:17.191012 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.191871 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.191872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:17.193187 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:17.193190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.194072 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.194072 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:17.194137 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:17.194147 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 14:57:17.194178 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 14:57:17.194219 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":45,"thread_start_us":98,"threads_started":1}
I20260928 14:57:17.194707 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:17.190094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:37177 (local address 127.0.0.1:47128)
0928 14:57:17.190504 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:17.190515 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:17.190596 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:17.190809 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:17.190810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:17.190812 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:17.190856 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:17.190857 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.192154 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.192155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:17.193049 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:17.193051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.194071 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.194072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:17.194137 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:17.194145 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 14:57:17.194194 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 14:57:17.194223 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":5,"thread_start_us":67,"threads_started":1}
I20260928 14:57:17.195621 7263 tablet_service.cc:1467] Processing CreateTablet for tablet a260267d99e34981bd25d2df9c5e21f4 (DEFAULT_TABLE table=test-table [id=28a9dd88ecb043cca5efcee51d705995]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:17.195633 7394 tablet_service.cc:1467] Processing CreateTablet for tablet a260267d99e34981bd25d2df9c5e21f4 (DEFAULT_TABLE table=test-table [id=28a9dd88ecb043cca5efcee51d705995]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:17.195962 7263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a260267d99e34981bd25d2df9c5e21f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:17.195969 7394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a260267d99e34981bd25d2df9c5e21f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:17.199079 7605 tablet_bootstrap.cc:492] T a260267d99e34981bd25d2df9c5e21f4 P d4e83b2c8e48473f8ad096af9fcc5fec: Bootstrap starting.
I20260928 14:57:17.199038 7525 tablet_service.cc:1467] Processing CreateTablet for tablet a260267d99e34981bd25d2df9c5e21f4 (DEFAULT_TABLE table=test-table [id=28a9dd88ecb043cca5efcee51d705995]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:17.199391 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a260267d99e34981bd25d2df9c5e21f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:17.199744 7605 tablet_bootstrap.cc:654] T a260267d99e34981bd25d2df9c5e21f4 P d4e83b2c8e48473f8ad096af9fcc5fec: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:17.200034 7605 log.cc:824] T a260267d99e34981bd25d2df9c5e21f4 P d4e83b2c8e48473f8ad096af9fcc5fec: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:17.200734 7605 tablet_bootstrap.cc:492] T a260267d99e34981bd25d2df9c5e21f4 P d4e83b2c8e48473f8ad096af9fcc5fec: No bootstrap required, opened a new log
I20260928 14:57:17.200793 7605 ts_tablet_manager.cc:1397] T a260267d99e34981bd25d2df9c5e21f4 P d4e83b2c8e48473f8ad096af9fcc5fec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:57:17.202234 7608 tablet_bootstrap.cc:492] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82: Bootstrap starting.
I20260928 14:57:17.202376 7606 tablet_bootstrap.cc:492] T a260267d99e34981bd25d2df9c5e21f4 P 73e432ec15a046e58097af104548da98: Bootstrap starting.
I20260928 14:57:17.203039 7606 tablet_bootstrap.cc:654] T a260267d99e34981bd25d2df9c5e21f4 P 73e432ec15a046e58097af104548da98: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:17.203039 7608 tablet_bootstrap.cc:654] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:17.203156 7605 raft_consensus.cc:348] T a260267d99e34981bd25d2df9c5e21f4 P d4e83b2c8e48473f8ad096af9fcc5fec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e432ec15a046e58097af104548da98" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38143 } } peers { permanent_uuid: "d4e83b2c8e48473f8ad096af9fcc5fec" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 33519 } } peers { permanent_uuid: "3f39d1907d8545e094d0e28b036edd82" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37177 } }
I20260928 14:57:17.203330 7605 raft_consensus.cc:374] T a260267d99e34981bd25d2df9c5e21f4 P d4e83b2c8e48473f8ad096af9fcc5fec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:17.203372 7605 raft_consensus.cc:729] T a260267d99e34981bd25d2df9c5e21f4 P d4e83b2c8e48473f8ad096af9fcc5fec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e83b2c8e48473f8ad096af9fcc5fec, State: Initialized, Role: FOLLOWER
I20260928 14:57:17.203377 7606 log.cc:824] T a260267d99e34981bd25d2df9c5e21f4 P 73e432ec15a046e58097af104548da98: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:17.203379 7608 log.cc:824] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:17.203526 7605 consensus_queue.cc:260] T a260267d99e34981bd25d2df9c5e21f4 P d4e83b2c8e48473f8ad096af9fcc5fec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e432ec15a046e58097af104548da98" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38143 } } peers { permanent_uuid: "d4e83b2c8e48473f8ad096af9fcc5fec" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 33519 } } peers { permanent_uuid: "3f39d1907d8545e094d0e28b036edd82" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37177 } }
I20260928 14:57:17.203831 7605 ts_tablet_manager.cc:1428] T a260267d99e34981bd25d2df9c5e21f4 P d4e83b2c8e48473f8ad096af9fcc5fec: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:57:17.203897 7460 heartbeater.cc:503] Master 127.20.116.190:43921 was elected leader, sending a full tablet report...
I20260928 14:57:17.204185 7606 tablet_bootstrap.cc:492] T a260267d99e34981bd25d2df9c5e21f4 P 73e432ec15a046e58097af104548da98: No bootstrap required, opened a new log
I20260928 14:57:17.204255 7606 ts_tablet_manager.cc:1397] T a260267d99e34981bd25d2df9c5e21f4 P 73e432ec15a046e58097af104548da98: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:57:17.204387 7608 tablet_bootstrap.cc:492] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82: No bootstrap required, opened a new log
I20260928 14:57:17.204461 7608 ts_tablet_manager.cc:1397] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:57:17.206804 7608 raft_consensus.cc:348] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e432ec15a046e58097af104548da98" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38143 } } peers { permanent_uuid: "d4e83b2c8e48473f8ad096af9fcc5fec" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 33519 } } peers { permanent_uuid: "3f39d1907d8545e094d0e28b036edd82" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37177 } }
I20260928 14:57:17.206943 7608 raft_consensus.cc:374] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:17.206841 7606 raft_consensus.cc:348] T a260267d99e34981bd25d2df9c5e21f4 P 73e432ec15a046e58097af104548da98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e432ec15a046e58097af104548da98" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38143 } } peers { permanent_uuid: "d4e83b2c8e48473f8ad096af9fcc5fec" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 33519 } } peers { permanent_uuid: "3f39d1907d8545e094d0e28b036edd82" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37177 } }
I20260928 14:57:17.206966 7608 raft_consensus.cc:729] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f39d1907d8545e094d0e28b036edd82, State: Initialized, Role: FOLLOWER
I20260928 14:57:17.206985 7606 raft_consensus.cc:374] T a260267d99e34981bd25d2df9c5e21f4 P 73e432ec15a046e58097af104548da98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:17.207041 7606 raft_consensus.cc:729] T a260267d99e34981bd25d2df9c5e21f4 P 73e432ec15a046e58097af104548da98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73e432ec15a046e58097af104548da98, State: Initialized, Role: FOLLOWER
I20260928 14:57:17.207093 7608 consensus_queue.cc:260] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e432ec15a046e58097af104548da98" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38143 } } peers { permanent_uuid: "d4e83b2c8e48473f8ad096af9fcc5fec" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 33519 } } peers { permanent_uuid: "3f39d1907d8545e094d0e28b036edd82" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37177 } }
I20260928 14:57:17.207144 7606 consensus_queue.cc:260] T a260267d99e34981bd25d2df9c5e21f4 P 73e432ec15a046e58097af104548da98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e432ec15a046e58097af104548da98" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38143 } } peers { permanent_uuid: "d4e83b2c8e48473f8ad096af9fcc5fec" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 33519 } } peers { permanent_uuid: "3f39d1907d8545e094d0e28b036edd82" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37177 } }
I20260928 14:57:17.207382 7608 ts_tablet_manager.cc:1428] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 14:57:17.207415 7591 heartbeater.cc:503] Master 127.20.116.190:43921 was elected leader, sending a full tablet report...
I20260928 14:57:17.207458 7606 ts_tablet_manager.cc:1428] T a260267d99e34981bd25d2df9c5e21f4 P 73e432ec15a046e58097af104548da98: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:57:17.207598 7329 heartbeater.cc:503] Master 127.20.116.190:43921 was elected leader, sending a full tablet report...
W20260928 14:57:17.286628 7461 tablet.cc:2369] T a260267d99e34981bd25d2df9c5e21f4 P d4e83b2c8e48473f8ad096af9fcc5fec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:57:17.346807 7612 raft_consensus.cc:482] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:57:17.346979 7612 raft_consensus.cc:504] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e432ec15a046e58097af104548da98" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38143 } } peers { permanent_uuid: "d4e83b2c8e48473f8ad096af9fcc5fec" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 33519 } } peers { permanent_uuid: "3f39d1907d8545e094d0e28b036edd82" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37177 } }
I20260928 14:57:17.347378 7612 leader_election.cc:288] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73e432ec15a046e58097af104548da98 (127.20.116.129:38143), d4e83b2c8e48473f8ad096af9fcc5fec (127.20.116.130:33519)
I20260928 14:57:17.350286 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:17.347660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:33083 (local address 127.20.116.129:38143)
0928 14:57:17.347730 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:17.347732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:17.347744 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:57:17.347794 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:17.347795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:17.347805 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:17.347889 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:17.348058 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.348577 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.349537 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.349927 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.350015 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:17.350026 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:17.350184 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 14:57:17.350221 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 14:57:17.350286 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:17.347583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:38143 (local address 127.20.116.131:33083)
0928 14:57:17.347656 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:17.347669 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:17.347720 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:17.347899 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:17.347900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:17.347902 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:17.347977 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:17.347980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.348670 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.348671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:17.349440 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:17.349442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.350018 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.350019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:17.350093 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:17.350100 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 14:57:17.350164 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 14:57:17.350209 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 14:57:17.350399 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:17.347583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:33519 (local address 127.20.116.131:48179)
0928 14:57:17.347913 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:17.347922 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:17.347986 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:17.348128 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:17.348129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:17.348131 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:17.348210 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:17.348211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.348841 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.348842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:17.349584 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:17.349586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.350128 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.350129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:17.350201 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:17.350204 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:57:17.350269 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 14:57:17.350309 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":111,"thread_start_us":159,"threads_started":1}
I20260928 14:57:17.350390 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:17.347672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:48179 (local address 127.20.116.130:33519)
0928 14:57:17.347724 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:17.347726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:17.347982 (+ 256us) server_negotiation.cc:407] Connection header received
0928 14:57:17.348047 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:17.348049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:17.348058 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:17.348124 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:17.348295 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.348757 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.349674 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.350045 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.350127 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:17.350135 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:17.350288 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 14:57:17.350329 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 14:57:17.350783 7283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260267d99e34981bd25d2df9c5e21f4" candidate_uuid: "3f39d1907d8545e094d0e28b036edd82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73e432ec15a046e58097af104548da98" is_pre_election: true
I20260928 14:57:17.350785 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260267d99e34981bd25d2df9c5e21f4" candidate_uuid: "3f39d1907d8545e094d0e28b036edd82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e83b2c8e48473f8ad096af9fcc5fec" is_pre_election: true
I20260928 14:57:17.350955 7283 raft_consensus.cc:2454] T a260267d99e34981bd25d2df9c5e21f4 P 73e432ec15a046e58097af104548da98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f39d1907d8545e094d0e28b036edd82 in term 0.
I20260928 14:57:17.350956 7414 raft_consensus.cc:2454] T a260267d99e34981bd25d2df9c5e21f4 P d4e83b2c8e48473f8ad096af9fcc5fec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f39d1907d8545e094d0e28b036edd82 in term 0.
I20260928 14:57:17.351198 7481 leader_election.cc:302] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f39d1907d8545e094d0e28b036edd82, d4e83b2c8e48473f8ad096af9fcc5fec; no voters:
I20260928 14:57:17.351382 7612 raft_consensus.cc:2781] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:57:17.351462 7612 raft_consensus.cc:482] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:57:17.351500 7612 raft_consensus.cc:3037] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:17.352236 7612 raft_consensus.cc:504] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e432ec15a046e58097af104548da98" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38143 } } peers { permanent_uuid: "d4e83b2c8e48473f8ad096af9fcc5fec" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 33519 } } peers { permanent_uuid: "3f39d1907d8545e094d0e28b036edd82" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37177 } }
I20260928 14:57:17.352394 7612 leader_election.cc:288] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82 [CANDIDATE]: Term 1 election: Requested vote from peers 73e432ec15a046e58097af104548da98 (127.20.116.129:38143), d4e83b2c8e48473f8ad096af9fcc5fec (127.20.116.130:33519)
I20260928 14:57:17.352550 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260267d99e34981bd25d2df9c5e21f4" candidate_uuid: "3f39d1907d8545e094d0e28b036edd82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e83b2c8e48473f8ad096af9fcc5fec"
I20260928 14:57:17.352566 7283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260267d99e34981bd25d2df9c5e21f4" candidate_uuid: "3f39d1907d8545e094d0e28b036edd82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73e432ec15a046e58097af104548da98"
I20260928 14:57:17.352653 7283 raft_consensus.cc:3037] T a260267d99e34981bd25d2df9c5e21f4 P 73e432ec15a046e58097af104548da98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:17.352653 7414 raft_consensus.cc:3037] T a260267d99e34981bd25d2df9c5e21f4 P d4e83b2c8e48473f8ad096af9fcc5fec [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:17.353596 7283 raft_consensus.cc:2454] T a260267d99e34981bd25d2df9c5e21f4 P 73e432ec15a046e58097af104548da98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f39d1907d8545e094d0e28b036edd82 in term 1.
I20260928 14:57:17.353596 7414 raft_consensus.cc:2454] T a260267d99e34981bd25d2df9c5e21f4 P d4e83b2c8e48473f8ad096af9fcc5fec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f39d1907d8545e094d0e28b036edd82 in term 1.
I20260928 14:57:17.353793 7481 leader_election.cc:302] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f39d1907d8545e094d0e28b036edd82, d4e83b2c8e48473f8ad096af9fcc5fec; no voters:
I20260928 14:57:17.353921 7612 raft_consensus.cc:2781] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:17.354110 7612 raft_consensus.cc:686] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82 [term 1 LEADER]: Becoming Leader. State: Replica: 3f39d1907d8545e094d0e28b036edd82, State: Running, Role: LEADER
I20260928 14:57:17.354211 7612 consensus_queue.cc:237] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e432ec15a046e58097af104548da98" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38143 } } peers { permanent_uuid: "d4e83b2c8e48473f8ad096af9fcc5fec" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 33519 } } peers { permanent_uuid: "3f39d1907d8545e094d0e28b036edd82" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37177 } }
I20260928 14:57:17.355181 7140 catalog_manager.cc:5495] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f39d1907d8545e094d0e28b036edd82 (127.20.116.131). New cstate: current_term: 1 leader_uuid: "3f39d1907d8545e094d0e28b036edd82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e432ec15a046e58097af104548da98" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4e83b2c8e48473f8ad096af9fcc5fec" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 33519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f39d1907d8545e094d0e28b036edd82" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 37177 } health_report { overall_health: HEALTHY } } }
I20260928 14:57:17.376380 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:17.373821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47142 (local address 127.20.116.131:37177)
0928 14:57:17.373882 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:17.373884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:17.373894 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:57:17.373943 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:17.373944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:17.373964 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:17.374041 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:17.374168 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.374699 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.375470 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:17.375673 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:17.375735 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:17.375907 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:17.375997 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:17.376075 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:17.376213 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 14:57:17.376233 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:57:17.378473 7414 raft_consensus.cc:1261] T a260267d99e34981bd25d2df9c5e21f4 P d4e83b2c8e48473f8ad096af9fcc5fec [term 1 FOLLOWER]: Refusing update from remote peer 3f39d1907d8545e094d0e28b036edd82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:17.378473 7283 raft_consensus.cc:1261] T a260267d99e34981bd25d2df9c5e21f4 P 73e432ec15a046e58097af104548da98 [term 1 FOLLOWER]: Refusing update from remote peer 3f39d1907d8545e094d0e28b036edd82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:17.378871 7615 consensus_queue.cc:1035] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73e432ec15a046e58097af104548da98" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 38143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:17.378988 7612 consensus_queue.cc:1035] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4e83b2c8e48473f8ad096af9fcc5fec" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 33519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:17.381458 7620 mvcc.cc:205] Tried to move back new op lower bound from 7334328063499124736 to 7334328063403630592. Current Snapshot: MvccSnapshot[applied={T|T < 7334328063499124736}]
I20260928 14:57:17.381294 7618 mvcc.cc:205] Tried to move back new op lower bound from 7334328063499124736 to 7334328063403630592. Current Snapshot: MvccSnapshot[applied={T|T < 7334328063499124736}]
I20260928 14:57:17.382057 7619 mvcc.cc:205] Tried to move back new op lower bound from 7334328063499124736 to 7334328063403630592. Current Snapshot: MvccSnapshot[applied={T|T < 7334328063499124736}]
I20260928 14:57:17.384387 7141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:17.384501 7141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:17.385252 7141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0338b2a995e24310a102c0dd3267669d: Sending DeleteTablet for 3 replicas of tablet a260267d99e34981bd25d2df9c5e21f4
I20260928 14:57:17.385607 7394 tablet_service.cc:1514] Processing DeleteTablet for tablet a260267d99e34981bd25d2df9c5e21f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:17 UTC) from {username='slave'} at 127.0.0.1:39426
I20260928 14:57:17.385653 7263 tablet_service.cc:1514] Processing DeleteTablet for tablet a260267d99e34981bd25d2df9c5e21f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:17 UTC) from {username='slave'} at 127.0.0.1:35356
I20260928 14:57:17.385867 7627 tablet_replica.cc:330] T a260267d99e34981bd25d2df9c5e21f4 P d4e83b2c8e48473f8ad096af9fcc5fec: stopping tablet replica
I20260928 14:57:17.385977 7627 raft_consensus.cc:2229] T a260267d99e34981bd25d2df9c5e21f4 P d4e83b2c8e48473f8ad096af9fcc5fec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:17.386085 7627 raft_consensus.cc:2258] T a260267d99e34981bd25d2df9c5e21f4 P d4e83b2c8e48473f8ad096af9fcc5fec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:17.386097 7525 tablet_service.cc:1514] Processing DeleteTablet for tablet a260267d99e34981bd25d2df9c5e21f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:17 UTC) from {username='slave'} at 127.0.0.1:47128
I20260928 14:57:17.386149 7628 tablet_replica.cc:330] T a260267d99e34981bd25d2df9c5e21f4 P 73e432ec15a046e58097af104548da98: stopping tablet replica
I20260928 14:57:17.386377 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 7201
I20260928 14:57:17.387076 7627 ts_tablet_manager.cc:1905] T a260267d99e34981bd25d2df9c5e21f4 P d4e83b2c8e48473f8ad096af9fcc5fec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:17.388918 7627 ts_tablet_manager.cc:1918] T a260267d99e34981bd25d2df9c5e21f4 P d4e83b2c8e48473f8ad096af9fcc5fec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:17.388996 7627 log.cc:1196] T a260267d99e34981bd25d2df9c5e21f4 P d4e83b2c8e48473f8ad096af9fcc5fec: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/a260267d99e34981bd25d2df9c5e21f4
I20260928 14:57:17.389268 7627 ts_tablet_manager.cc:1939] T a260267d99e34981bd25d2df9c5e21f4 P d4e83b2c8e48473f8ad096af9fcc5fec: Deleting consensus metadata
I20260928 14:57:17.389730 7128 catalog_manager.cc:4841] TS d4e83b2c8e48473f8ad096af9fcc5fec (127.20.116.130:33519): tablet a260267d99e34981bd25d2df9c5e21f4 (table test-table [id=28a9dd88ecb043cca5efcee51d705995]) successfully deleted
I20260928 14:57:17.390667 7629 tablet_replica.cc:330] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82: stopping tablet replica
I20260928 14:57:17.390820 7629 raft_consensus.cc:2229] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:57:17.390973 7629 raft_consensus.cc:2258] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:17.391393 7629 ts_tablet_manager.cc:1905] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:17.393417 7629 ts_tablet_manager.cc:1918] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:17.393509 7629 log.cc:1196] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/a260267d99e34981bd25d2df9c5e21f4
W20260928 14:57:17.393775 7127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.116.129:38143 (error 108)
I20260928 14:57:17.393976 7629 ts_tablet_manager.cc:1939] T a260267d99e34981bd25d2df9c5e21f4 P 3f39d1907d8545e094d0e28b036edd82: Deleting consensus metadata
I20260928 14:57:17.394011 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 7332
I20260928 14:57:17.395246 7127 catalog_manager.cc:4841] TS 3f39d1907d8545e094d0e28b036edd82 (127.20.116.131:37177): tablet a260267d99e34981bd25d2df9c5e21f4 (table test-table [id=28a9dd88ecb043cca5efcee51d705995]) successfully deleted
I20260928 14:57:17.399096 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:17.398933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:38143 (local address 127.0.0.1:35370)
0928 14:57:17.398996 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:17.399055 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.129:38143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 14:57:17.399245 7127 catalog_manager.cc:4584] TS 73e432ec15a046e58097af104548da98 (127.20.116.129:38143): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a260267d99e34981bd25d2df9c5e21f4: Network error: Client connection negotiation failed: client connection to 127.20.116.129:38143: connect: Connection refused (error 111)
I20260928 14:57:17.401628 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 7463
I20260928 14:57:17.406729 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 7110
2026-09-28T14:57:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (803 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:17 dist-test-slave-pxl6 krb5kdc[7633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:17 dist-test-slave-pxl6 krb5kdc[7633](info): set up 2 sockets
Sep 28 14:57:17 dist-test-slave-pxl6 krb5kdc[7633](info): commencing operation
krb5kdc: starting...
W20260928 14:57:18.475594 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:18 dist-test-slave-pxl6 krb5kdc[7633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:57:18.601951 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:33943
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:33943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:18.685909 7646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:18.686127 7646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:18.686151 7646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:18.687287 7646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:18.687325 7646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:18.687340 7646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:18.687352 7646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:18.687364 7646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:18.688818 7646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:33943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:33943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:18.689031 7646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:18.689205 7646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:18.694240 7651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:18.694262 7652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:18.694288 7646 server_base.cc:1034] running on GCE node
W20260928 14:57:18.694264 7654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:18.694849 7646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:18.695178 7646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:18.696333 7646 hybrid_clock.cc:648] HybridClock initialized: now 1790607438696323 us; error 27 us; skew 500 ppm
Sep 28 14:57:18 dist-test-slave-pxl6 krb5kdc[7633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607438, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:18.698840 7646 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:57:18.699290 7646 webserver.cc:460] Webserver started at http://127.20.116.190:35127/ using document root <none> and password file <none>
I20260928 14:57:18.699481 7646 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:18.699522 7646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:18.699611 7646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:18.700626 7646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "6ec53d87847c435cacabceb1aa01257e"
format_stamp: "Formatted at 2026-09-28 14:57:18 on dist-test-slave-pxl6"
server_key: "e128a4d245dfb0f5c2a49d32978f30bd"
server_key_iv: "b8bab23dd4c074643ad9ad8f19e13887"
server_key_version: "encryptionkey@0"
I20260928 14:57:18.700955 7646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "6ec53d87847c435cacabceb1aa01257e"
format_stamp: "Formatted at 2026-09-28 14:57:18 on dist-test-slave-pxl6"
server_key: "e128a4d245dfb0f5c2a49d32978f30bd"
server_key_iv: "b8bab23dd4c074643ad9ad8f19e13887"
server_key_version: "encryptionkey@0"
I20260928 14:57:18.702184 7646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 14:57:18.702946 7661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:18.703107 7646 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:18.703212 7646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "6ec53d87847c435cacabceb1aa01257e"
format_stamp: "Formatted at 2026-09-28 14:57:18 on dist-test-slave-pxl6"
server_key: "e128a4d245dfb0f5c2a49d32978f30bd"
server_key_iv: "b8bab23dd4c074643ad9ad8f19e13887"
server_key_version: "encryptionkey@0"
I20260928 14:57:18.703277 7646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:18.712672 7646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:18.714372 7646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:18.714561 7646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:18.718447 7646 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:33943
I20260928 14:57:18.718504 7713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:33943 every 8 connection(s)
I20260928 14:57:18.718895 7646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:18.719751 7714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:18.721918 7714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e: Bootstrap starting.
I20260928 14:57:18.722435 7714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:18.722744 7714 log.cc:824] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:18.723384 7714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e: No bootstrap required, opened a new log
I20260928 14:57:18.724942 7714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec53d87847c435cacabceb1aa01257e" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 33943 } }
I20260928 14:57:18.725076 7714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:18.725097 7714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ec53d87847c435cacabceb1aa01257e, State: Initialized, Role: FOLLOWER
I20260928 14:57:18.725232 7714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec53d87847c435cacabceb1aa01257e" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 33943 } }
I20260928 14:57:18.725314 7714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:18.725338 7714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:18.725376 7714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:18.725848 7714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec53d87847c435cacabceb1aa01257e" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 33943 } }
I20260928 14:57:18.725930 7714 leader_election.cc:302] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ec53d87847c435cacabceb1aa01257e; no voters:
I20260928 14:57:18.726061 7714 leader_election.cc:288] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:18.726125 7717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:18.726271 7717 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e [term 1 LEADER]: Becoming Leader. State: Replica: 6ec53d87847c435cacabceb1aa01257e, State: Running, Role: LEADER
I20260928 14:57:18.726302 7714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:18.726367 7717 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec53d87847c435cacabceb1aa01257e" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 33943 } }
I20260928 14:57:18.726929 7719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ec53d87847c435cacabceb1aa01257e. Latest consensus state: current_term: 1 leader_uuid: "6ec53d87847c435cacabceb1aa01257e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec53d87847c435cacabceb1aa01257e" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 33943 } } }
I20260928 14:57:18.727029 7719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:18.727033 7718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ec53d87847c435cacabceb1aa01257e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec53d87847c435cacabceb1aa01257e" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 33943 } } }
I20260928 14:57:18.727154 7718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:18.727351 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 7646
I20260928 14:57:18.727396 7726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:18.727469 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:18.727717 20946 external_mini_cluster.cc:1428] Setting key cb028ef86ff59adfe88eb718bda51a97
I20260928 14:57:18.728000 7726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:18.729897 7726 catalog_manager.cc:1347] Generated new cluster ID: 7e465aff79a349ed9cb0186b7fbdb2f4
I20260928 14:57:18.729949 7726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:57:18 dist-test-slave-pxl6 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:57:18.736407 7726 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:18.737145 7726 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:18.739617 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:18.728352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60202 (local address 127.20.116.190:33943)
0928 14:57:18.728765 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:18.728773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:18.728800 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:57:18.728888 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:18.728893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:18.728945 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:18.729082 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:18.729850 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:18.731261 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:18.732312 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:18.732981 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:18.733086 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:18.735843 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:18.735846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:18.735847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:18.735859 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:18.737989 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:18.738398 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:18.738401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:18.738402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:18.738516 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:18.738733 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:18.738733 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:18.738734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:18.738987 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:18.739145 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:18.739249 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 14:57:18.739384 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":79,"threads_started":1}
I20260928 14:57:18.742329 7726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e: Generated new TSK 0
I20260928 14:57:18.742508 7726 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
Entry for principal kudu/127.20.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab.
I20260928 14:57:18.771785 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:33943
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:18.856317 7740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:18.856523 7740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:18.856547 7740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:18.857700 7740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:18.857739 7740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:18.857767 7740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:57:18.859212 7740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.129.keytab
--principal=kudu/127.20.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:33943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:18.859407 7740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:18.859606 7740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:18.864634 7745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:18.864638 7748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:18.864856 7740 server_base.cc:1034] running on GCE node
W20260928 14:57:18.864686 7746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:18.865162 7740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:18.865422 7740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:18.866595 7740 hybrid_clock.cc:648] HybridClock initialized: now 1790607438866580 us; error 30 us; skew 500 ppm
Sep 28 14:57:18 dist-test-slave-pxl6 krb5kdc[7633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607438, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:18.869321 7740 init.cc:377] Logged in from keytab as kudu/127.20.116.129@KRBTEST.COM (short username kudu)
I20260928 14:57:18.869781 7740 webserver.cc:460] Webserver started at http://127.20.116.129:34397/ using document root <none> and password file <none>
I20260928 14:57:18.870012 7740 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:18.870059 7740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:18.870174 7740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:18.871418 7740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "3f4f4a4f74224ca2a2c3f4449fc3c031"
format_stamp: "Formatted at 2026-09-28 14:57:18 on dist-test-slave-pxl6"
server_key: "36c443935380641b3772719ad1bc4dcd"
server_key_iv: "029b4c14f37325c7dd8d3e793d0d9a19"
server_key_version: "encryptionkey@0"
I20260928 14:57:18.871768 7740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "3f4f4a4f74224ca2a2c3f4449fc3c031"
format_stamp: "Formatted at 2026-09-28 14:57:18 on dist-test-slave-pxl6"
server_key: "36c443935380641b3772719ad1bc4dcd"
server_key_iv: "029b4c14f37325c7dd8d3e793d0d9a19"
server_key_version: "encryptionkey@0"
I20260928 14:57:18.873208 7740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:57:18.873980 7755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:18.874194 7740 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:57:18.874298 7740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "3f4f4a4f74224ca2a2c3f4449fc3c031"
format_stamp: "Formatted at 2026-09-28 14:57:18 on dist-test-slave-pxl6"
server_key: "36c443935380641b3772719ad1bc4dcd"
server_key_iv: "029b4c14f37325c7dd8d3e793d0d9a19"
server_key_version: "encryptionkey@0"
I20260928 14:57:18.874378 7740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:18.916898 7740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:18.919297 7740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:18.919462 7740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:18.919713 7740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:18.920054 7740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:18.920089 7740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:18.920112 7740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:18.920125 7740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:18.926108 7740 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:34727
I20260928 14:57:18.926170 7868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:34727 every 8 connection(s)
I20260928 14:57:18.926509 7740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:57:18.928476 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 7740
I20260928 14:57:18.928596 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:57:18.928781 20946 external_mini_cluster.cc:1428] Setting key 1cee69b979aa4e311d585bb0fb9667e7
Sep 28 14:57:18 dist-test-slave-pxl6 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607438, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.129@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:57:18.937419 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:18.927473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:33031 (local address 127.20.116.190:33943)
0928 14:57:18.927529 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:18.927531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:18.927782 (+ 251us) server_negotiation.cc:407] Connection header received
0928 14:57:18.928459 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:18.928461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:18.928476 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:18.928562 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:18.929586 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:18.930369 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:18.931881 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:18.932112 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:18.932185 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:18.934467 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:18.934468 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:18.934469 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:18.934477 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:18.935983 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:18.936382 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:18.936383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:18.936383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:18.936446 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:18.936699 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:18.936700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:18.936700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:18.936836 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:18.936939 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:18.937218 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 14:57:18.937323 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:57:18.937669 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:18.927479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:33943 (local address 127.20.116.129:33031)
0928 14:57:18.927655 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:18.927694 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:18.928339 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:18.928628 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:18.928631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:18.929283 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:18.929489 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:18.929495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:18.930455 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:18.930456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:18.931743 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:18.931746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:18.932214 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:18.932215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:18.932357 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:18.932968 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:18.932977 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:18.934365 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:18.936065 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:18.936073 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:18.936075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:18.936309 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:18.936511 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:18.936512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:18.936512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:18.936633 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:18.936930 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:18.936937 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:18.937086 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 14:57:18.937194 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 14:57:18.937362 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260928 14:57:18.938126 7869 heartbeater.cc:347] Connected to a master server at 127.20.116.190:33943
WARNING: no policy specified for kudu/127.20.116.130@KRBTEST.COM; defaulting to no policy
I20260928 14:57:18.938223 7869 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:18.938423 7869 heartbeater.cc:511] Master 127.20.116.190:33943 requested a full tablet report, sending...
I20260928 14:57:18.938980 7678 ts_manager.cc:194] Registered new tserver with Master: 3f4f4a4f74224ca2a2c3f4449fc3c031 (127.20.116.129:34727)
I20260928 14:57:18.939683 7678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.129@KRBTEST.COM'} at 127.20.116.129:33031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
Entry for principal kudu/127.20.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab.
I20260928 14:57:18.957049 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:33943
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:19.039418 7874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:19.039630 7874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:19.039680 7874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:19.040864 7874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:19.040907 7874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:19.040933 7874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:57:19.042411 7874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.130.keytab
--principal=kudu/127.20.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:33943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:19.042694 7874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:19.042927 7874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:19.048007 7882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:19.047983 7880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:19.047982 7879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:19.048242 7874 server_base.cc:1034] running on GCE node
I20260928 14:57:19.048446 7874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:19.048708 7874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:19.049877 7874 hybrid_clock.cc:648] HybridClock initialized: now 1790607439049871 us; error 35 us; skew 500 ppm
Sep 28 14:57:19 dist-test-slave-pxl6 krb5kdc[7633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607439, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:19.052477 7874 init.cc:377] Logged in from keytab as kudu/127.20.116.130@KRBTEST.COM (short username kudu)
I20260928 14:57:19.052910 7874 webserver.cc:460] Webserver started at http://127.20.116.130:45547/ using document root <none> and password file <none>
I20260928 14:57:19.053216 7874 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:19.053288 7874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:19.053431 7874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:19.054509 7874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "c2003fd8590b4538bb7e2deb77d519b6"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-pxl6"
server_key: "ec8ed41d3853ed01c96dbc5ddfffc249"
server_key_iv: "b52afabdf410e2d4d820220c623b4648"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.054942 7874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "c2003fd8590b4538bb7e2deb77d519b6"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-pxl6"
server_key: "ec8ed41d3853ed01c96dbc5ddfffc249"
server_key_iv: "b52afabdf410e2d4d820220c623b4648"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.056258 7874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:57:19.057003 7889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:19.057173 7874 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:57:19.057271 7874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "c2003fd8590b4538bb7e2deb77d519b6"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-pxl6"
server_key: "ec8ed41d3853ed01c96dbc5ddfffc249"
server_key_iv: "b52afabdf410e2d4d820220c623b4648"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.057349 7874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:19.079164 7874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:19.081764 7874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:19.081940 7874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:19.082221 7874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:19.082589 7874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:19.082645 7874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:19.082682 7874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:19.082708 7874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:19.088549 7874 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:36517
I20260928 14:57:19.088598 8002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:36517 every 8 connection(s)
I20260928 14:57:19.088936 7874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:57:19.092404 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 7874
I20260928 14:57:19.092517 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:57:19.092717 20946 external_mini_cluster.cc:1428] Setting key c6a4fe371279c72be3479677f5d5e863
Sep 28 14:57:19 dist-test-slave-pxl6 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607439, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.130@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:57:19.099787 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.089629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:35879 (local address 127.20.116.190:33943)
0928 14:57:19.089718 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:19.089721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:19.089941 (+ 220us) server_negotiation.cc:407] Connection header received
0928 14:57:19.090452 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:19.090453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:19.090464 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:19.090517 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:19.091518 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.092009 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.093561 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.093782 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.093850 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.096675 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:19.096677 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:19.096678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:19.096687 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:19.098307 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:19.098793 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:19.098795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:19.098795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:19.098854 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:19.099117 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:19.099118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:19.099118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:19.099237 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:19.099317 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:19.099595 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 14:57:19.099683 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:57:19.099972 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.089604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:33943 (local address 127.20.116.130:35879)
0928 14:57:19.089810 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:19.089851 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:19.090367 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:19.090648 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:19.090651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:19.091224 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:19.091439 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:19.091444 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.092089 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.092090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:19.093331 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:19.093334 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.093873 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.093875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:19.094039 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.094634 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:19.094646 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:19.096558 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:19.098387 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:19.098396 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:19.098399 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:19.098704 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:19.098918 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:19.098919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:19.098919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:19.099044 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:19.099306 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:19.099313 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:19.099460 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 14:57:19.099561 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 14:57:19.099728 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":107,"threads_started":1}
I20260928 14:57:19.100514 8003 heartbeater.cc:347] Connected to a master server at 127.20.116.190:33943
I20260928 14:57:19.100639 8003 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:19.100854 8003 heartbeater.cc:511] Master 127.20.116.190:33943 requested a full tablet report, sending...
I20260928 14:57:19.101272 7678 ts_manager.cc:194] Registered new tserver with Master: c2003fd8590b4538bb7e2deb77d519b6 (127.20.116.130:36517)
WARNING: no policy specified for kudu/127.20.116.131@KRBTEST.COM; defaulting to no policy
I20260928 14:57:19.101707 7678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.130@KRBTEST.COM'} at 127.20.116.130:35879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
Entry for principal kudu/127.20.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab.
I20260928 14:57:19.123215 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:33943
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:33609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:19.213569 8008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:19.213778 8008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:19.213801 8008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:19.214958 8008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:19.215000 8008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:19.215027 8008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:57:19.216514 8008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:33609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.131.keytab
--principal=kudu/127.20.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:33943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:19.216747 8008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:19.216936 8008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:19.222043 8014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:19.222038 8013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:19.222041 8016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:19.222077 8008 server_base.cc:1034] running on GCE node
I20260928 14:57:19.222436 8008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:19.222823 8008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:19.224006 8008 hybrid_clock.cc:648] HybridClock initialized: now 1790607439223992 us; error 23 us; skew 500 ppm
Sep 28 14:57:19 dist-test-slave-pxl6 krb5kdc[7633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607439, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:19.226758 8008 init.cc:377] Logged in from keytab as kudu/127.20.116.131@KRBTEST.COM (short username kudu)
I20260928 14:57:19.227312 8008 webserver.cc:460] Webserver started at http://127.20.116.131:40821/ using document root <none> and password file <none>
I20260928 14:57:19.227562 8008 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:19.227633 8008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:19.227767 8008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:19.228847 8008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "c39a108fb797475b89802fc55f8b113a"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-pxl6"
server_key: "0cb012fdba00bf85ac8df882a5fdfbb4"
server_key_iv: "0942e89201b2f62ba96df12b6451ab8f"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.229202 8008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "c39a108fb797475b89802fc55f8b113a"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-pxl6"
server_key: "0cb012fdba00bf85ac8df882a5fdfbb4"
server_key_iv: "0942e89201b2f62ba96df12b6451ab8f"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.230522 8008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:19.231344 8023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:19.231503 8008 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:57:19.231637 8008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "c39a108fb797475b89802fc55f8b113a"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-pxl6"
server_key: "0cb012fdba00bf85ac8df882a5fdfbb4"
server_key_iv: "0942e89201b2f62ba96df12b6451ab8f"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.231714 8008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:19.250805 8008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:19.253106 8008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:19.253289 8008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:19.253536 8008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:19.253875 8008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:19.253926 8008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:19.253968 8008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:19.254001 8008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:19.260324 8008 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:38447
I20260928 14:57:19.260434 8136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:38447 every 8 connection(s)
I20260928 14:57:19.260795 8008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:57:19 dist-test-slave-pxl6 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607439, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.131@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:57:19.269204 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 8008
I20260928 14:57:19.269315 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:57:19.269526 20946 external_mini_cluster.cc:1428] Setting key 269a38d7902a95af86a7d2a88fd7d19e
I20260928 14:57:19.271313 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.261642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:49523 (local address 127.20.116.190:33943)
0928 14:57:19.261717 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:19.261719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:19.261943 (+ 224us) server_negotiation.cc:407] Connection header received
0928 14:57:19.262559 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:19.262560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:19.262574 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:19.262624 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:19.263673 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.264193 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.265238 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.265453 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.265535 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.267894 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:19.267896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:19.267896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:19.267904 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:19.269569 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:19.270128 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:19.270130 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:19.270130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:19.270218 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:19.270620 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:19.270621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:19.270621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:19.270787 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:19.270843 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:19.271176 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 14:57:19.271249 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:57:19.271548 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.261631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:33943 (local address 127.20.116.131:49523)
0928 14:57:19.261817 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:19.261856 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:19.262411 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:19.262700 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:19.262705 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:19.263317 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:19.263543 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:19.263548 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.264298 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.264299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:19.265133 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:19.265135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.265555 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.265556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:19.265693 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.266197 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:19.266206 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:19.267788 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:19.269688 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:19.269698 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:19.269701 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:19.269978 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:19.270309 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:19.270310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:19.270310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:19.270459 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:19.270867 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:19.270874 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:19.271029 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 14:57:19.271129 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 14:57:19.271282 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260928 14:57:19.272058 8137 heartbeater.cc:347] Connected to a master server at 127.20.116.190:33943
I20260928 14:57:19.272154 8137 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:19.272341 8137 heartbeater.cc:511] Master 127.20.116.190:33943 requested a full tablet report, sending...
I20260928 14:57:19.272701 7678 ts_manager.cc:194] Registered new tserver with Master: c39a108fb797475b89802fc55f8b113a (127.20.116.131:38447)
I20260928 14:57:19.273096 7678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.116.131@KRBTEST.COM'} at 127.20.116.131:49523
I20260928 14:57:19.282096 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:57:19.290019 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.283697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60206 (local address 127.20.116.190:33943)
0928 14:57:19.283771 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:19.283774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:19.283864 (+ 90us) server_negotiation.cc:407] Connection header received
0928 14:57:19.283956 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:19.283958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:19.283969 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:19.284029 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:19.284588 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.285092 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.285832 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.286045 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.286129 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.287471 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:19.287472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:19.287472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:19.287480 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:19.288798 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:19.289167 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:19.289168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:19.289169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:19.289267 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:19.289463 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:19.289464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:19.289464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:19.289625 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:19.289730 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:19.289830 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 14:57:19.289912 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:57:19.291599 7678 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:57:19.293108 7678 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:57:19.299870 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.296107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38748 (local address 127.20.116.131:38447)
0928 14:57:19.296430 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:19.296435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:19.296444 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:57:19.296482 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:19.296484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:19.296500 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:19.296594 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:19.296701 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.297731 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.298928 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.299474 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.299568 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.299628 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:19.299748 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 14:57:19.299791 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":84,"threads_started":1}
I20260928 14:57:19.299858 8149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.295884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:38447 (local address 127.0.0.1:38748)
0928 14:57:19.296228 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:19.296237 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:19.296273 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:19.296598 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:19.296599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:19.296601 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:19.296636 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:19.296637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.297820 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.297821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:19.298829 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:19.298832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.299569 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.299570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:19.299646 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.299660 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 14:57:19.299728 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 14:57:19.299780 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260928 14:57:19.300086 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.295970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33478 (local address 127.20.116.130:36517)
0928 14:57:19.296182 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:19.296187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:19.296199 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:57:19.296238 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:19.296240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:19.296256 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:19.296521 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:19.296622 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.297334 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.298798 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.299521 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.299592 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.299661 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:19.299916 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 14:57:19.299961 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":96,"threads_started":1}
I20260928 14:57:19.300107 8151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.295577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:36517 (local address 127.0.0.1:33478)
0928 14:57:19.296105 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:19.296115 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:19.296160 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:19.296386 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:19.296388 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:19.296390 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:19.296442 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:19.296444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.297430 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.297431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:19.298705 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:19.298708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.299698 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.299699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:19.299765 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.299770 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:57:19.299826 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 14:57:19.299869 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":4,"thread_start_us":104,"threads_started":1}
I20260928 14:57:19.300611 8148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.295576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:34727 (local address 127.0.0.1:48368)
0928 14:57:19.295737 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:19.295818 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:19.295887 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:19.296300 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:19.296305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:19.296307 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:19.296386 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:19.296387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.297791 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.297792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:19.298995 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:19.298998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.300384 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.300385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:19.300446 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.300450 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:57:19.300507 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 14:57:19.300545 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":99,"threads_started":1}
I20260928 14:57:19.300686 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.295729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.20.116.129:34727)
0928 14:57:19.295938 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:19.295943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:19.295954 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:57:19.295998 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:19.296001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:19.296017 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:19.296135 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:19.296508 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.297672 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.299077 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.299782 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.299843 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.299887 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:19.300550 (+ 663us) server_negotiation.cc:299] Negotiation successful
0928 14:57:19.300598 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":99,"threads_started":1}
I20260928 14:57:19.300632 8071 tablet_service.cc:1467] Processing CreateTablet for tablet b7c698ccb2484de5a4a04b9d3b6e7d72 (DEFAULT_TABLE table=test-table [id=cf561f33ebc34d1c8da294239d752a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:19.300779 7937 tablet_service.cc:1467] Processing CreateTablet for tablet b7c698ccb2484de5a4a04b9d3b6e7d72 (DEFAULT_TABLE table=test-table [id=cf561f33ebc34d1c8da294239d752a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:19.301016 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c698ccb2484de5a4a04b9d3b6e7d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:19.301116 7937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c698ccb2484de5a4a04b9d3b6e7d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:19.301632 7803 tablet_service.cc:1467] Processing CreateTablet for tablet b7c698ccb2484de5a4a04b9d3b6e7d72 (DEFAULT_TABLE table=test-table [id=cf561f33ebc34d1c8da294239d752a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:19.301936 7803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c698ccb2484de5a4a04b9d3b6e7d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:19.303802 8154 tablet_bootstrap.cc:492] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6: Bootstrap starting.
I20260928 14:57:19.304446 8155 tablet_bootstrap.cc:492] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c39a108fb797475b89802fc55f8b113a: Bootstrap starting.
I20260928 14:57:19.304540 8154 tablet_bootstrap.cc:654] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:19.305162 8155 tablet_bootstrap.cc:654] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c39a108fb797475b89802fc55f8b113a: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:19.305163 8156 tablet_bootstrap.cc:492] T b7c698ccb2484de5a4a04b9d3b6e7d72 P 3f4f4a4f74224ca2a2c3f4449fc3c031: Bootstrap starting.
I20260928 14:57:19.305280 8154 log.cc:824] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:19.305820 8156 tablet_bootstrap.cc:654] T b7c698ccb2484de5a4a04b9d3b6e7d72 P 3f4f4a4f74224ca2a2c3f4449fc3c031: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:19.305805 8155 log.cc:824] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c39a108fb797475b89802fc55f8b113a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:19.306106 8154 tablet_bootstrap.cc:492] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6: No bootstrap required, opened a new log
I20260928 14:57:19.306197 8154 ts_tablet_manager.cc:1397] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:57:19.306375 8156 log.cc:824] T b7c698ccb2484de5a4a04b9d3b6e7d72 P 3f4f4a4f74224ca2a2c3f4449fc3c031: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:19.306629 8155 tablet_bootstrap.cc:492] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c39a108fb797475b89802fc55f8b113a: No bootstrap required, opened a new log
I20260928 14:57:19.306700 8155 ts_tablet_manager.cc:1397] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c39a108fb797475b89802fc55f8b113a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:57:19.307087 8156 tablet_bootstrap.cc:492] T b7c698ccb2484de5a4a04b9d3b6e7d72 P 3f4f4a4f74224ca2a2c3f4449fc3c031: No bootstrap required, opened a new log
I20260928 14:57:19.307199 8156 ts_tablet_manager.cc:1397] T b7c698ccb2484de5a4a04b9d3b6e7d72 P 3f4f4a4f74224ca2a2c3f4449fc3c031: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:57:19.308234 8154 raft_consensus.cc:348] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2003fd8590b4538bb7e2deb77d519b6" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 36517 } } peers { permanent_uuid: "3f4f4a4f74224ca2a2c3f4449fc3c031" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34727 } } peers { permanent_uuid: "c39a108fb797475b89802fc55f8b113a" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38447 } }
I20260928 14:57:19.308427 8154 raft_consensus.cc:374] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:19.308465 8154 raft_consensus.cc:729] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2003fd8590b4538bb7e2deb77d519b6, State: Initialized, Role: FOLLOWER
I20260928 14:57:19.308630 8154 consensus_queue.cc:260] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2003fd8590b4538bb7e2deb77d519b6" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 36517 } } peers { permanent_uuid: "3f4f4a4f74224ca2a2c3f4449fc3c031" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34727 } } peers { permanent_uuid: "c39a108fb797475b89802fc55f8b113a" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38447 } }
I20260928 14:57:19.308918 8154 ts_tablet_manager.cc:1428] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 14:57:19.308930 8003 heartbeater.cc:503] Master 127.20.116.190:33943 was elected leader, sending a full tablet report...
I20260928 14:57:19.308843 8156 raft_consensus.cc:348] T b7c698ccb2484de5a4a04b9d3b6e7d72 P 3f4f4a4f74224ca2a2c3f4449fc3c031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2003fd8590b4538bb7e2deb77d519b6" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 36517 } } peers { permanent_uuid: "3f4f4a4f74224ca2a2c3f4449fc3c031" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34727 } } peers { permanent_uuid: "c39a108fb797475b89802fc55f8b113a" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38447 } }
I20260928 14:57:19.308820 8155 raft_consensus.cc:348] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c39a108fb797475b89802fc55f8b113a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2003fd8590b4538bb7e2deb77d519b6" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 36517 } } peers { permanent_uuid: "3f4f4a4f74224ca2a2c3f4449fc3c031" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34727 } } peers { permanent_uuid: "c39a108fb797475b89802fc55f8b113a" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38447 } }
I20260928 14:57:19.309018 8156 raft_consensus.cc:374] T b7c698ccb2484de5a4a04b9d3b6e7d72 P 3f4f4a4f74224ca2a2c3f4449fc3c031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:19.309036 8155 raft_consensus.cc:374] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c39a108fb797475b89802fc55f8b113a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:19.309051 8156 raft_consensus.cc:729] T b7c698ccb2484de5a4a04b9d3b6e7d72 P 3f4f4a4f74224ca2a2c3f4449fc3c031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f4f4a4f74224ca2a2c3f4449fc3c031, State: Initialized, Role: FOLLOWER
I20260928 14:57:19.309069 8155 raft_consensus.cc:729] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c39a108fb797475b89802fc55f8b113a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39a108fb797475b89802fc55f8b113a, State: Initialized, Role: FOLLOWER
I20260928 14:57:19.309165 8156 consensus_queue.cc:260] T b7c698ccb2484de5a4a04b9d3b6e7d72 P 3f4f4a4f74224ca2a2c3f4449fc3c031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2003fd8590b4538bb7e2deb77d519b6" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 36517 } } peers { permanent_uuid: "3f4f4a4f74224ca2a2c3f4449fc3c031" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34727 } } peers { permanent_uuid: "c39a108fb797475b89802fc55f8b113a" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38447 } }
I20260928 14:57:19.309208 8155 consensus_queue.cc:260] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c39a108fb797475b89802fc55f8b113a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2003fd8590b4538bb7e2deb77d519b6" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 36517 } } peers { permanent_uuid: "3f4f4a4f74224ca2a2c3f4449fc3c031" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34727 } } peers { permanent_uuid: "c39a108fb797475b89802fc55f8b113a" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38447 } }
I20260928 14:57:19.309432 8156 ts_tablet_manager.cc:1428] T b7c698ccb2484de5a4a04b9d3b6e7d72 P 3f4f4a4f74224ca2a2c3f4449fc3c031: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:57:19.309510 8155 ts_tablet_manager.cc:1428] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c39a108fb797475b89802fc55f8b113a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:57:19.309571 7869 heartbeater.cc:503] Master 127.20.116.190:33943 was elected leader, sending a full tablet report...
I20260928 14:57:19.309631 8137 heartbeater.cc:503] Master 127.20.116.190:33943 was elected leader, sending a full tablet report...
W20260928 14:57:19.339514 8004 tablet.cc:2369] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:57:19.385190 8160 raft_consensus.cc:482] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:57:19.385346 8160 raft_consensus.cc:504] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2003fd8590b4538bb7e2deb77d519b6" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 36517 } } peers { permanent_uuid: "3f4f4a4f74224ca2a2c3f4449fc3c031" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34727 } } peers { permanent_uuid: "c39a108fb797475b89802fc55f8b113a" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38447 } }
I20260928 14:57:19.385726 8160 leader_election.cc:288] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f4f4a4f74224ca2a2c3f4449fc3c031 (127.20.116.129:34727), c39a108fb797475b89802fc55f8b113a (127.20.116.131:38447)
I20260928 14:57:19.388759 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.385997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:60929 (local address 127.20.116.129:34727)
0928 14:57:19.386052 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:19.386055 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:19.386067 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:57:19.386149 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:19.386150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:19.386166 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:19.386229 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:19.386411 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.387025 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.388009 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.388389 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.388446 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.388453 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:19.388624 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 14:57:19.388674 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 14:57:19.388767 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.386058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:33959 (local address 127.20.116.131:38447)
0928 14:57:19.386104 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:19.386106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:19.386230 (+ 124us) server_negotiation.cc:407] Connection header received
0928 14:57:19.386310 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:19.386312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:19.386327 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:19.386391 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:19.386568 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.387038 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.388037 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.388409 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.388496 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.388504 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:19.388661 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 14:57:19.388700 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:57:19.388767 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.385944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:34727 (local address 127.20.116.130:60929)
0928 14:57:19.386009 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:19.386021 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:19.386070 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:19.386245 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:19.386247 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:19.386249 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:19.386328 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:19.386330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.387106 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.387108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:19.387911 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:19.387914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.388462 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.388463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:19.388546 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.388549 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:57:19.388601 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 14:57:19.388659 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":21}
I20260928 14:57:19.388775 8163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.385957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:38447 (local address 127.20.116.130:33959)
0928 14:57:19.386133 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:19.386143 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:19.386214 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:19.386409 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:19.386410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:19.386412 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:19.386479 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:19.386481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.387144 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.387145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:19.387928 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:19.387930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.388514 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.388514 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:19.388570 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.388573 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:57:19.388644 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 14:57:19.388678 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":110,"threads_started":1}
I20260928 14:57:19.389158 7823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c698ccb2484de5a4a04b9d3b6e7d72" candidate_uuid: "c2003fd8590b4538bb7e2deb77d519b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f4f4a4f74224ca2a2c3f4449fc3c031" is_pre_election: true
I20260928 14:57:19.389353 7823 raft_consensus.cc:2454] T b7c698ccb2484de5a4a04b9d3b6e7d72 P 3f4f4a4f74224ca2a2c3f4449fc3c031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2003fd8590b4538bb7e2deb77d519b6 in term 0.
I20260928 14:57:19.389423 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c698ccb2484de5a4a04b9d3b6e7d72" candidate_uuid: "c2003fd8590b4538bb7e2deb77d519b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39a108fb797475b89802fc55f8b113a" is_pre_election: true
I20260928 14:57:19.389596 8091 raft_consensus.cc:2454] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c39a108fb797475b89802fc55f8b113a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2003fd8590b4538bb7e2deb77d519b6 in term 0.
I20260928 14:57:19.389618 7891 leader_election.cc:302] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f4f4a4f74224ca2a2c3f4449fc3c031, c2003fd8590b4538bb7e2deb77d519b6; no voters:
I20260928 14:57:19.389775 8160 raft_consensus.cc:2781] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:57:19.389838 8160 raft_consensus.cc:482] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:57:19.389873 8160 raft_consensus.cc:3037] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:19.390588 8160 raft_consensus.cc:504] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2003fd8590b4538bb7e2deb77d519b6" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 36517 } } peers { permanent_uuid: "3f4f4a4f74224ca2a2c3f4449fc3c031" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34727 } } peers { permanent_uuid: "c39a108fb797475b89802fc55f8b113a" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38447 } }
I20260928 14:57:19.390748 8160 leader_election.cc:288] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6 [CANDIDATE]: Term 1 election: Requested vote from peers 3f4f4a4f74224ca2a2c3f4449fc3c031 (127.20.116.129:34727), c39a108fb797475b89802fc55f8b113a (127.20.116.131:38447)
I20260928 14:57:19.390885 7823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c698ccb2484de5a4a04b9d3b6e7d72" candidate_uuid: "c2003fd8590b4538bb7e2deb77d519b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f4f4a4f74224ca2a2c3f4449fc3c031"
I20260928 14:57:19.390901 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c698ccb2484de5a4a04b9d3b6e7d72" candidate_uuid: "c2003fd8590b4538bb7e2deb77d519b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39a108fb797475b89802fc55f8b113a"
I20260928 14:57:19.390960 7823 raft_consensus.cc:3037] T b7c698ccb2484de5a4a04b9d3b6e7d72 P 3f4f4a4f74224ca2a2c3f4449fc3c031 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:19.390980 8091 raft_consensus.cc:3037] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c39a108fb797475b89802fc55f8b113a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:19.391690 8091 raft_consensus.cc:2454] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c39a108fb797475b89802fc55f8b113a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2003fd8590b4538bb7e2deb77d519b6 in term 1.
I20260928 14:57:19.391705 7823 raft_consensus.cc:2454] T b7c698ccb2484de5a4a04b9d3b6e7d72 P 3f4f4a4f74224ca2a2c3f4449fc3c031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2003fd8590b4538bb7e2deb77d519b6 in term 1.
I20260928 14:57:19.391870 7890 leader_election.cc:302] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2003fd8590b4538bb7e2deb77d519b6, c39a108fb797475b89802fc55f8b113a; no voters:
I20260928 14:57:19.391997 8160 raft_consensus.cc:2781] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:19.392161 8160 raft_consensus.cc:686] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6 [term 1 LEADER]: Becoming Leader. State: Replica: c2003fd8590b4538bb7e2deb77d519b6, State: Running, Role: LEADER
I20260928 14:57:19.392246 8160 consensus_queue.cc:237] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2003fd8590b4538bb7e2deb77d519b6" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 36517 } } peers { permanent_uuid: "3f4f4a4f74224ca2a2c3f4449fc3c031" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34727 } } peers { permanent_uuid: "c39a108fb797475b89802fc55f8b113a" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38447 } }
I20260928 14:57:19.393077 7678 catalog_manager.cc:5495] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2003fd8590b4538bb7e2deb77d519b6 (127.20.116.130). New cstate: current_term: 1 leader_uuid: "c2003fd8590b4538bb7e2deb77d519b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2003fd8590b4538bb7e2deb77d519b6" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 36517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f4f4a4f74224ca2a2c3f4449fc3c031" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c39a108fb797475b89802fc55f8b113a" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38447 } health_report { overall_health: UNKNOWN } } }
I20260928 14:57:19.419200 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.416659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33480 (local address 127.20.116.130:36517)
0928 14:57:19.416728 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:19.416730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:19.416737 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:57:19.416801 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:19.416802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:19.416823 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:19.416899 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:19.417058 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.417538 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.418326 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.418570 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.418628 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.418831 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:19.418948 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:19.419041 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:19.419059 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 14:57:19.419080 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:57:19.421233 8091 raft_consensus.cc:1261] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c39a108fb797475b89802fc55f8b113a [term 1 FOLLOWER]: Refusing update from remote peer c2003fd8590b4538bb7e2deb77d519b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:19.421233 7823 raft_consensus.cc:1261] T b7c698ccb2484de5a4a04b9d3b6e7d72 P 3f4f4a4f74224ca2a2c3f4449fc3c031 [term 1 FOLLOWER]: Refusing update from remote peer c2003fd8590b4538bb7e2deb77d519b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:19.421607 8160 consensus_queue.cc:1035] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39a108fb797475b89802fc55f8b113a" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 38447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:19.421744 8164 consensus_queue.cc:1035] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f4f4a4f74224ca2a2c3f4449fc3c031" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 34727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:19.424202 8167 mvcc.cc:205] Tried to move back new op lower bound from 7334328071866068992 to 7334328071751352320. Current Snapshot: MvccSnapshot[applied={T|T < 7334328071866068992}]
I20260928 14:57:19.427105 7678 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:19.427211 7678 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:19.428092 7678 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ec53d87847c435cacabceb1aa01257e: Sending DeleteTablet for 3 replicas of tablet b7c698ccb2484de5a4a04b9d3b6e7d72
I20260928 14:57:19.428438 7937 tablet_service.cc:1514] Processing DeleteTablet for tablet b7c698ccb2484de5a4a04b9d3b6e7d72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:19 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:33478
I20260928 14:57:19.428499 7803 tablet_service.cc:1514] Processing DeleteTablet for tablet b7c698ccb2484de5a4a04b9d3b6e7d72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:19 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:48368
I20260928 14:57:19.428511 8071 tablet_service.cc:1514] Processing DeleteTablet for tablet b7c698ccb2484de5a4a04b9d3b6e7d72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:19 UTC) from {username='kudu', principal='kudu/127.20.116.190@KRBTEST.COM'} at 127.0.0.1:38748
I20260928 14:57:19.428691 8176 tablet_replica.cc:330] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6: stopping tablet replica
I20260928 14:57:19.428810 8176 raft_consensus.cc:2229] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:57:19.428947 8176 raft_consensus.cc:2258] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:19.429059 8178 tablet_replica.cc:330] T b7c698ccb2484de5a4a04b9d3b6e7d72 P 3f4f4a4f74224ca2a2c3f4449fc3c031: stopping tablet replica
I20260928 14:57:19.429136 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 7740
I20260928 14:57:19.429173 8178 raft_consensus.cc:2229] T b7c698ccb2484de5a4a04b9d3b6e7d72 P 3f4f4a4f74224ca2a2c3f4449fc3c031 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:19.429277 8178 raft_consensus.cc:2258] T b7c698ccb2484de5a4a04b9d3b6e7d72 P 3f4f4a4f74224ca2a2c3f4449fc3c031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:19.429719 8177 tablet_replica.cc:330] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c39a108fb797475b89802fc55f8b113a: stopping tablet replica
I20260928 14:57:19.429845 8177 raft_consensus.cc:2229] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c39a108fb797475b89802fc55f8b113a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:19.429950 8177 raft_consensus.cc:2258] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c39a108fb797475b89802fc55f8b113a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:19.430476 8177 ts_tablet_manager.cc:1905] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c39a108fb797475b89802fc55f8b113a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:19.432185 8176 ts_tablet_manager.cc:1905] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:19.432374 8177 ts_tablet_manager.cc:1918] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c39a108fb797475b89802fc55f8b113a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:19.432461 8177 log.cc:1196] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c39a108fb797475b89802fc55f8b113a: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/b7c698ccb2484de5a4a04b9d3b6e7d72
I20260928 14:57:19.432744 8177 ts_tablet_manager.cc:1939] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c39a108fb797475b89802fc55f8b113a: Deleting consensus metadata
I20260928 14:57:19.433198 7662 catalog_manager.cc:4841] TS c39a108fb797475b89802fc55f8b113a (127.20.116.131:38447): tablet b7c698ccb2484de5a4a04b9d3b6e7d72 (table test-table [id=cf561f33ebc34d1c8da294239d752a3b]) successfully deleted
I20260928 14:57:19.434376 8176 ts_tablet_manager.cc:1918] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:19.434464 8176 log.cc:1196] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/b7c698ccb2484de5a4a04b9d3b6e7d72
I20260928 14:57:19.434851 8176 ts_tablet_manager.cc:1939] T b7c698ccb2484de5a4a04b9d3b6e7d72 P c2003fd8590b4538bb7e2deb77d519b6: Deleting consensus metadata
I20260928 14:57:19.435580 7662 catalog_manager.cc:4841] TS c2003fd8590b4538bb7e2deb77d519b6 (127.20.116.130:36517): tablet b7c698ccb2484de5a4a04b9d3b6e7d72 (table test-table [id=cf561f33ebc34d1c8da294239d752a3b]) successfully deleted
W20260928 14:57:19.436992 7663 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.116.129:34727 (error 108)
I20260928 14:57:19.437264 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 7874
I20260928 14:57:19.439131 8148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.438964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:34727 (local address 127.0.0.1:48378)
0928 14:57:19.439020 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:19.439066 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.129:34727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 14:57:19.439275 7663 catalog_manager.cc:4584] TS 3f4f4a4f74224ca2a2c3f4449fc3c031 (127.20.116.129:34727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b7c698ccb2484de5a4a04b9d3b6e7d72: Network error: Client connection negotiation failed: client connection to 127.20.116.129:34727: connect: Connection refused (error 111)
I20260928 14:57:19.442818 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 8008
I20260928 14:57:19.448503 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 7646
2026-09-28T14:57:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2043 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T14:57:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:19Z Disabled control of system clock
I20260928 14:57:19.477030 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:40611
--webserver_interface=127.20.116.190
--webserver_port=0
--builtin_ntp_servers=127.20.116.148:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:40611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:57:19.559423 8183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:19.559656 8183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:19.559679 8183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:19.560956 8183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:19.560997 8183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:19.561010 8183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:19.561023 8183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:19.561035 8183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:19.562367 8183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:40611
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:40611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:19.562638 8183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:19.562911 8183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:19.568022 8188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:19.568037 8191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:19.568123 8183 server_base.cc:1034] running on GCE node
W20260928 14:57:19.568045 8189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:19.568678 8183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:19.569005 8183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:19.570150 8183 hybrid_clock.cc:648] HybridClock initialized: now 1790607439570129 us; error 35 us; skew 500 ppm
I20260928 14:57:19.571547 8183 webserver.cc:460] Webserver started at http://127.20.116.190:40683/ using document root <none> and password file <none>
I20260928 14:57:19.571820 8183 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:19.571889 8183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:19.572032 8183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:19.573110 8183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "2552a1fafdb34b5fb59e58c86ed969f4"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-pxl6"
server_key: "c2cad5dbdcafcd56fdf95f1ec7c7f06d"
server_key_iv: "5bb16d0ef8c3085d2399a5d27a8f470b"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.573469 8183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "2552a1fafdb34b5fb59e58c86ed969f4"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-pxl6"
server_key: "c2cad5dbdcafcd56fdf95f1ec7c7f06d"
server_key_iv: "5bb16d0ef8c3085d2399a5d27a8f470b"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.574829 8183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:57:19.575556 8197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:19.575732 8183 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:57:19.575835 8183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "2552a1fafdb34b5fb59e58c86ed969f4"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-pxl6"
server_key: "c2cad5dbdcafcd56fdf95f1ec7c7f06d"
server_key_iv: "5bb16d0ef8c3085d2399a5d27a8f470b"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.575903 8183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:19.587916 8183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:19.588214 8183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:19.588362 8183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:19.592411 8183 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:40611
I20260928 14:57:19.592449 8249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:40611 every 8 connection(s)
I20260928 14:57:19.592795 8183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:19.593786 8250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:19.596192 8250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4: Bootstrap starting.
I20260928 14:57:19.596800 8250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:19.597108 8250 log.cc:824] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:19.597777 8250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4: No bootstrap required, opened a new log
I20260928 14:57:19.599489 8250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2552a1fafdb34b5fb59e58c86ed969f4" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40611 } }
I20260928 14:57:19.599632 8250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:19.599675 8250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2552a1fafdb34b5fb59e58c86ed969f4, State: Initialized, Role: FOLLOWER
I20260928 14:57:19.599826 8250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2552a1fafdb34b5fb59e58c86ed969f4" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40611 } }
I20260928 14:57:19.599887 8250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:19.599936 8250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:19.599985 8250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:19.600510 8250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2552a1fafdb34b5fb59e58c86ed969f4" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40611 } }
I20260928 14:57:19.600628 8250 leader_election.cc:302] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2552a1fafdb34b5fb59e58c86ed969f4; no voters:
I20260928 14:57:19.600818 8250 leader_election.cc:288] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:19.600881 8253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:19.601071 8253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4 [term 1 LEADER]: Becoming Leader. State: Replica: 2552a1fafdb34b5fb59e58c86ed969f4, State: Running, Role: LEADER
I20260928 14:57:19.601111 8250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:19.601174 8253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2552a1fafdb34b5fb59e58c86ed969f4" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40611 } }
I20260928 14:57:19.601456 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 8183
I20260928 14:57:19.601553 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:19.601755 20946 external_mini_cluster.cc:1428] Setting key e8e0fff1f685e77cd7d37534ededda47
I20260928 14:57:19.602396 8255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2552a1fafdb34b5fb59e58c86ed969f4. Latest consensus state: current_term: 1 leader_uuid: "2552a1fafdb34b5fb59e58c86ed969f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2552a1fafdb34b5fb59e58c86ed969f4" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40611 } } }
I20260928 14:57:19.602486 8255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:19.602732 8254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2552a1fafdb34b5fb59e58c86ed969f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2552a1fafdb34b5fb59e58c86ed969f4" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40611 } } }
I20260928 14:57:19.602807 8269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:19.602871 8254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:19.603340 8269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:19.604939 8269 catalog_manager.cc:1347] Generated new cluster ID: 4e181ed5f2f84c0b910d2a50905ceb91
I20260928 14:57:19.604995 8269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:19.608278 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.602295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.20.116.190:40611)
0928 14:57:19.602597 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:19.602608 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:19.602808 (+ 200us) server_negotiation.cc:407] Connection header received
0928 14:57:19.603211 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:19.603219 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:19.603293 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:19.603694 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:19.603792 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.605017 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.606124 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.606748 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.606862 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.607577 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:19.607581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:19.607581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:19.607784 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:19.607826 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:19.607839 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:19.607901 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:19.607994 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 14:57:19.608072 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260928 14:57:19.617205 8269 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:19.617928 8269 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:19.627040 8269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4: Generated new TSK 0
I20260928 14:57:19.627259 8269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:19.632514 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.129:0
--local_ip_for_outbound_sockets=127.20.116.129
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:40611
--builtin_ntp_servers=127.20.116.148:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:57:19.721261 8274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:19.721496 8274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:19.721534 8274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:19.722785 8274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:19.722844 8274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:19.722888 8274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.129
I20260928 14:57:19.724323 8274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.116.129
--webserver_port=0
--tserver_master_addrs=127.20.116.190:40611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.129
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:19.724615 8274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:19.724924 8274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:19.731175 8280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:19.731191 8282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:19.731429 8274 server_base.cc:1034] running on GCE node
W20260928 14:57:19.731237 8279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:19.731714 8274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:19.731986 8274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:19.733145 8274 hybrid_clock.cc:648] HybridClock initialized: now 1790607439733126 us; error 35 us; skew 500 ppm
I20260928 14:57:19.734419 8274 webserver.cc:460] Webserver started at http://127.20.116.129:39961/ using document root <none> and password file <none>
I20260928 14:57:19.734727 8274 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:19.734776 8274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:19.734920 8274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:19.735993 8274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-0/data/instance:
uuid: "bc2a3929a7804f97a09327e5fd735851"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-pxl6"
server_key: "22623731394b5b07abd403c422068a91"
server_key_iv: "d90ec1974c538df92259ccbe033ff219"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.736379 8274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance:
uuid: "bc2a3929a7804f97a09327e5fd735851"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-pxl6"
server_key: "22623731394b5b07abd403c422068a91"
server_key_iv: "d90ec1974c538df92259ccbe033ff219"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.737722 8274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:19.738502 8288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:19.738694 8274 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:57:19.738794 8274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-0/wal
uuid: "bc2a3929a7804f97a09327e5fd735851"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-pxl6"
server_key: "22623731394b5b07abd403c422068a91"
server_key_iv: "d90ec1974c538df92259ccbe033ff219"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.738864 8274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:19.749989 8274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:19.750311 8274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:19.750459 8274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:19.750715 8274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:19.751053 8274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:19.751104 8274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:19.751147 8274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:19.751181 8274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:19.756928 8274 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.129:41199
I20260928 14:57:19.757012 8401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.129:41199 every 8 connection(s)
I20260928 14:57:19.757349 8274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-0/data/info.pb
I20260928 14:57:19.758420 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 8274
I20260928 14:57:19.758512 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-0/wal/instance
I20260928 14:57:19.758769 20946 external_mini_cluster.cc:1428] Setting key 08481d1b1361712d81fe29ee082ca0bb
I20260928 14:57:19.759783 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.130:0
--local_ip_for_outbound_sockets=127.20.116.130
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:40611
--builtin_ntp_servers=127.20.116.148:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:57:19.763067 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.758227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.129:38501 (local address 127.20.116.190:40611)
0928 14:57:19.758301 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:19.758304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:19.758575 (+ 271us) server_negotiation.cc:407] Connection header received
0928 14:57:19.759210 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:19.759211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:19.759222 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:19.759266 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:19.760064 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.760589 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.761605 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.761815 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.761887 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.762774 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:19.762776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:19.762776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:19.762783 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:19.762799 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:19.762803 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:19.762853 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:19.762948 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 14:57:19.762984 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:57:19.763314 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.758183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40611 (local address 127.20.116.129:38501)
0928 14:57:19.758391 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:19.758434 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:19.759113 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:19.759342 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:19.759347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:19.759358 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:19.759969 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:19.759976 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.760690 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.760691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:19.761513 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:19.761516 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.761910 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.761912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:19.762047 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.762620 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:57:19.762620 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:19.762651 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:57:19.762674 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:19.762858 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:19.762864 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:19.762867 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:57:19.762952 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 14:57:19.763061 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":105,"threads_started":1}
I20260928 14:57:19.763756 8402 heartbeater.cc:347] Connected to a master server at 127.20.116.190:40611
I20260928 14:57:19.763851 8402 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:19.764058 8402 heartbeater.cc:511] Master 127.20.116.190:40611 requested a full tablet report, sending...
I20260928 14:57:19.764590 8214 ts_manager.cc:194] Registered new tserver with Master: bc2a3929a7804f97a09327e5fd735851 (127.20.116.129:41199)
I20260928 14:57:19.765221 8214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.116.129:38501
W20260928 14:57:19.842799 8405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:19.843036 8405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:19.843075 8405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:19.844308 8405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:19.844362 8405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:19.844404 8405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.130
I20260928 14:57:19.845788 8405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.116.130
--webserver_port=0
--tserver_master_addrs=127.20.116.190:40611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.130
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:19.846062 8405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:19.846401 8405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:19.851593 8410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:19.851626 8411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:19.851652 8405 server_base.cc:1034] running on GCE node
W20260928 14:57:19.851593 8413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:19.852124 8405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:19.852391 8405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:19.853528 8405 hybrid_clock.cc:648] HybridClock initialized: now 1790607439853521 us; error 31 us; skew 500 ppm
I20260928 14:57:19.854879 8405 webserver.cc:460] Webserver started at http://127.20.116.130:43615/ using document root <none> and password file <none>
I20260928 14:57:19.855100 8405 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:19.855144 8405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:19.855237 8405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:19.856331 8405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-1/data/instance:
uuid: "3e10aaab448e49399c885c9e35bbefa7"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-pxl6"
server_key: "1f22fcb3381c5cadffab5be08fe0c734"
server_key_iv: "c14130ceec9a31216a40bd0b6676e427"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.856637 8405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance:
uuid: "3e10aaab448e49399c885c9e35bbefa7"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-pxl6"
server_key: "1f22fcb3381c5cadffab5be08fe0c734"
server_key_iv: "c14130ceec9a31216a40bd0b6676e427"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.857883 8405 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:57:19.858639 8419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:19.858812 8405 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:57:19.858902 8405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-1/wal
uuid: "3e10aaab448e49399c885c9e35bbefa7"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-pxl6"
server_key: "1f22fcb3381c5cadffab5be08fe0c734"
server_key_iv: "c14130ceec9a31216a40bd0b6676e427"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.858975 8405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:19.877532 8405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:19.877892 8405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:19.878055 8405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:19.878324 8405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:19.878757 8405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:19.878813 8405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:19.878856 8405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:19.878887 8405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:19.885047 8405 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.130:38407
I20260928 14:57:19.885103 8532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.130:38407 every 8 connection(s)
I20260928 14:57:19.885456 8405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-1/data/info.pb
I20260928 14:57:19.890906 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.886190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.130:39067 (local address 127.20.116.190:40611)
0928 14:57:19.886241 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:19.886243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:19.886561 (+ 318us) server_negotiation.cc:407] Connection header received
0928 14:57:19.887091 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:19.887093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:19.887107 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:19.887182 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:19.887878 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.888403 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.889368 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.889564 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.889669 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.890647 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:19.890649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:19.890649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:19.890655 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:19.890675 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:19.890679 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:19.890727 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:19.890818 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 14:57:19.890845 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:57:19.891149 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:19.886157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40611 (local address 127.20.116.130:39067)
0928 14:57:19.886372 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:19.886415 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:19.886977 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:19.887254 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:19.887258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:19.887267 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:19.887789 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:19.887793 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.888483 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.888484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:19.889292 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:19.889294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:19.889688 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:19.889689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:19.889819 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:19.890409 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:57:19.890409 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:19.890440 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:57:19.890463 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:19.890740 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:19.890746 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:19.890748 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:57:19.890810 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 14:57:19.890896 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":91,"threads_started":1}
I20260928 14:57:19.891705 8533 heartbeater.cc:347] Connected to a master server at 127.20.116.190:40611
I20260928 14:57:19.891829 8533 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:19.892082 8533 heartbeater.cc:511] Master 127.20.116.190:40611 requested a full tablet report, sending...
I20260928 14:57:19.892488 8214 ts_manager.cc:194] Registered new tserver with Master: 3e10aaab448e49399c885c9e35bbefa7 (127.20.116.130:38407)
I20260928 14:57:19.892843 8214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.116.130:39067
I20260928 14:57:19.895021 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 8405
I20260928 14:57:19.895154 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-1/wal/instance
I20260928 14:57:19.895362 20946 external_mini_cluster.cc:1428] Setting key 3508d69912367687d58171caa5caed1e
I20260928 14:57:19.896274 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.116.131:0
--local_ip_for_outbound_sockets=127.20.116.131
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:40611
--builtin_ntp_servers=127.20.116.148:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:57:19.980970 8536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:19.981191 8536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:19.981213 8536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:19.982350 8536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:19.982385 8536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:19.982410 8536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.116.131
I20260928 14:57:19.983943 8536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.116.131
--webserver_port=0
--tserver_master_addrs=127.20.116.190:40611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.116.131
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:19.984162 8536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:19.984426 8536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:19.989568 8544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:19.989573 8542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:19.989748 8536 server_base.cc:1034] running on GCE node
W20260928 14:57:19.989862 8541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:19.990119 8536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:19.990371 8536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:19.991546 8536 hybrid_clock.cc:648] HybridClock initialized: now 1790607439991537 us; error 37 us; skew 500 ppm
I20260928 14:57:19.992981 8536 webserver.cc:460] Webserver started at http://127.20.116.131:43101/ using document root <none> and password file <none>
I20260928 14:57:19.993297 8536 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:19.993368 8536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:19.993510 8536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:19.994789 8536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-2/data/instance:
uuid: "b9094f10591f4c5cbfc35893eecd660c"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-pxl6"
server_key: "afe44f1bdfddc5ee45ebc504798352c0"
server_key_iv: "61ae2147410c25cf7c570df7be3e378e"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.995254 8536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance:
uuid: "b9094f10591f4c5cbfc35893eecd660c"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-pxl6"
server_key: "afe44f1bdfddc5ee45ebc504798352c0"
server_key_iv: "61ae2147410c25cf7c570df7be3e378e"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.996742 8536 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:57:19.997532 8550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:19.997714 8536 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:57:19.997822 8536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-2/wal
uuid: "b9094f10591f4c5cbfc35893eecd660c"
format_stamp: "Formatted at 2026-09-28 14:57:19 on dist-test-slave-pxl6"
server_key: "afe44f1bdfddc5ee45ebc504798352c0"
server_key_iv: "61ae2147410c25cf7c570df7be3e378e"
server_key_version: "encryptionkey@0"
I20260928 14:57:19.997900 8536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:20.008710 8536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:20.009104 8536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:20.009262 8536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:20.009516 8536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:20.009888 8536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:20.009943 8536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:20.009985 8536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:20.010010 8536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:20.016052 8536 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.131:36969
I20260928 14:57:20.016093 8663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.131:36969 every 8 connection(s)
I20260928 14:57:20.016461 8536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-2/data/info.pb
I20260928 14:57:20.021392 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 8536
I20260928 14:57:20.021516 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-2/wal/instance
I20260928 14:57:20.021752 20946 external_mini_cluster.cc:1428] Setting key 85ce6531f5f7efc46fc1ef2e53a978ea
I20260928 14:57:20.022359 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:20.017174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:53403 (local address 127.20.116.190:40611)
0928 14:57:20.017242 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:20.017245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:20.017561 (+ 316us) server_negotiation.cc:407] Connection header received
0928 14:57:20.018069 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:20.018071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:20.018085 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:20.018145 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:20.018962 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.019518 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.020533 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.020778 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.020841 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:20.021950 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:20.021952 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:20.021952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:20.021959 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:20.021980 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:20.021985 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:20.022045 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:20.022205 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 14:57:20.022261 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 14:57:20.022679 8666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:20.017181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:40611 (local address 127.20.116.131:53403)
0928 14:57:20.017391 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:20.017455 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:20.017974 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:20.018201 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:20.018205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:20.018220 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:20.018851 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:20.018856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.019617 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.019618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:20.020435 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:20.020437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.020860 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.020861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:20.020972 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:20.021764 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:57:20.021764 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:20.021810 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:57:20.021841 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:20.022088 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:20.022098 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:20.022103 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:57:20.022207 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 14:57:20.022364 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260928 14:57:20.023222 8664 heartbeater.cc:347] Connected to a master server at 127.20.116.190:40611
I20260928 14:57:20.023327 8664 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:20.023519 8664 heartbeater.cc:511] Master 127.20.116.190:40611 requested a full tablet report, sending...
I20260928 14:57:20.023941 8214 ts_manager.cc:194] Registered new tserver with Master: b9094f10591f4c5cbfc35893eecd660c (127.20.116.131:36969)
I20260928 14:57:20.024369 8214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.116.131:53403
I20260928 14:57:20.034124 20946 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:57:20.038616 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:20.035689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52900 (local address 127.20.116.190:40611)
0928 14:57:20.035754 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:20.035756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:20.035892 (+ 136us) server_negotiation.cc:407] Connection header received
0928 14:57:20.035937 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:20.035937 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:20.035950 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:20.036010 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:20.036143 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.036636 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.037366 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.037546 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.037598 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:20.038372 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:20.038372 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:20.038373 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:20.038377 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:20.038388 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:20.038391 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:20.038427 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:20.038466 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 14:57:20.038489 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:57:20.040313 8214 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:57:20.041931 8214 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:57:20.048476 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:20.044217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47614 (local address 127.20.116.131:36969)
0928 14:57:20.044411 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:20.044416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:20.044427 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:57:20.044469 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:20.044470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:20.044482 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:20.044572 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:20.044829 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.045870 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.047090 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.047854 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.048144 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:20.048256 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:20.048338 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 14:57:20.048380 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":113,"threads_started":1}
I20260928 14:57:20.048463 8672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:20.044120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.131:36969 (local address 127.0.0.1:47614)
0928 14:57:20.044256 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:20.044296 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:20.044356 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:20.044666 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:20.044669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:20.044671 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:20.044724 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:20.044725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.045950 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.045951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:20.046989 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:20.046992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.048148 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.048149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:20.048230 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:20.048243 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 14:57:20.048324 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 14:57:20.048379 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":89,"threads_started":1}
I20260928 14:57:20.048713 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:20.044561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:41199 (local address 127.0.0.1:43360)
0928 14:57:20.044766 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:20.044776 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:20.044832 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:20.045260 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:20.045261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:20.045263 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:20.045310 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:20.045311 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.046673 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.046674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:20.047585 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:20.047589 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.048452 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.048453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:20.048518 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:20.048522 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:57:20.048564 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 14:57:20.048615 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":59,"threads_started":1}
I20260928 14:57:20.049160 8676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:20.044667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43360 (local address 127.20.116.129:41199)
0928 14:57:20.045082 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:20.045086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:20.045096 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:57:20.045135 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:20.045137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:20.045150 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:20.045255 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:20.045382 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.046525 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.047681 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.048347 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.048893 (+ 546us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:20.049001 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:20.049030 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 14:57:20.049072 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":87,"threads_started":1}
I20260928 14:57:20.049221 8598 tablet_service.cc:1467] Processing CreateTablet for tablet 68ddb4ed8f7a4488b88c682996e0de63 (DEFAULT_TABLE table=test-table [id=de87734593a74375a1641696bc217438]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:20.049636 8598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ddb4ed8f7a4488b88c682996e0de63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:20.049798 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:20.044371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:38407 (local address 127.0.0.1:41826)
0928 14:57:20.044497 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:20.044505 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:20.044543 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:20.045378 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:20.045378 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:20.045380 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:20.045404 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:20.045405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.046723 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.046724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:20.047823 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:20.047827 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.049549 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.049550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:20.049618 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:20.049621 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:57:20.049685 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 14:57:20.049727 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260928 14:57:20.049823 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:20.044863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41826 (local address 127.20.116.130:38407)
0928 14:57:20.045046 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:20.045051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:20.045063 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:57:20.045110 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:20.045112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:20.045138 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:20.045255 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:20.045480 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.046645 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.047921 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.049187 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.049551 (+ 364us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:20.049646 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:20.049701 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 14:57:20.049741 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":86,"threads_started":1}
I20260928 14:57:20.049806 8336 tablet_service.cc:1467] Processing CreateTablet for tablet 68ddb4ed8f7a4488b88c682996e0de63 (DEFAULT_TABLE table=test-table [id=de87734593a74375a1641696bc217438]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:20.050125 8336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ddb4ed8f7a4488b88c682996e0de63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:20.050433 8467 tablet_service.cc:1467] Processing CreateTablet for tablet 68ddb4ed8f7a4488b88c682996e0de63 (DEFAULT_TABLE table=test-table [id=de87734593a74375a1641696bc217438]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:20.050814 8467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ddb4ed8f7a4488b88c682996e0de63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:20.053370 8678 tablet_bootstrap.cc:492] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c: Bootstrap starting.
I20260928 14:57:20.053428 8679 tablet_bootstrap.cc:492] T 68ddb4ed8f7a4488b88c682996e0de63 P 3e10aaab448e49399c885c9e35bbefa7: Bootstrap starting.
I20260928 14:57:20.054055 8679 tablet_bootstrap.cc:654] T 68ddb4ed8f7a4488b88c682996e0de63 P 3e10aaab448e49399c885c9e35bbefa7: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:20.054056 8678 tablet_bootstrap.cc:654] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:20.054359 8679 log.cc:824] T 68ddb4ed8f7a4488b88c682996e0de63 P 3e10aaab448e49399c885c9e35bbefa7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:20.054363 8678 log.cc:824] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:20.054706 8680 tablet_bootstrap.cc:492] T 68ddb4ed8f7a4488b88c682996e0de63 P bc2a3929a7804f97a09327e5fd735851: Bootstrap starting.
I20260928 14:57:20.055187 8678 tablet_bootstrap.cc:492] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c: No bootstrap required, opened a new log
I20260928 14:57:20.055271 8678 ts_tablet_manager.cc:1397] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 14:57:20.055306 8680 tablet_bootstrap.cc:654] T 68ddb4ed8f7a4488b88c682996e0de63 P bc2a3929a7804f97a09327e5fd735851: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:20.055373 8679 tablet_bootstrap.cc:492] T 68ddb4ed8f7a4488b88c682996e0de63 P 3e10aaab448e49399c885c9e35bbefa7: No bootstrap required, opened a new log
I20260928 14:57:20.055444 8679 ts_tablet_manager.cc:1397] T 68ddb4ed8f7a4488b88c682996e0de63 P 3e10aaab448e49399c885c9e35bbefa7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 14:57:20.055629 8680 log.cc:824] T 68ddb4ed8f7a4488b88c682996e0de63 P bc2a3929a7804f97a09327e5fd735851: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:20.056452 8680 tablet_bootstrap.cc:492] T 68ddb4ed8f7a4488b88c682996e0de63 P bc2a3929a7804f97a09327e5fd735851: No bootstrap required, opened a new log
I20260928 14:57:20.056522 8680 ts_tablet_manager.cc:1397] T 68ddb4ed8f7a4488b88c682996e0de63 P bc2a3929a7804f97a09327e5fd735851: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:57:20.057689 8678 raft_consensus.cc:348] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9094f10591f4c5cbfc35893eecd660c" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36969 } } peers { permanent_uuid: "3e10aaab448e49399c885c9e35bbefa7" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 38407 } } peers { permanent_uuid: "bc2a3929a7804f97a09327e5fd735851" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41199 } }
I20260928 14:57:20.057853 8678 raft_consensus.cc:374] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:20.057898 8678 raft_consensus.cc:729] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9094f10591f4c5cbfc35893eecd660c, State: Initialized, Role: FOLLOWER
I20260928 14:57:20.058022 8678 consensus_queue.cc:260] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9094f10591f4c5cbfc35893eecd660c" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36969 } } peers { permanent_uuid: "3e10aaab448e49399c885c9e35bbefa7" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 38407 } } peers { permanent_uuid: "bc2a3929a7804f97a09327e5fd735851" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41199 } }
I20260928 14:57:20.058147 8679 raft_consensus.cc:348] T 68ddb4ed8f7a4488b88c682996e0de63 P 3e10aaab448e49399c885c9e35bbefa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9094f10591f4c5cbfc35893eecd660c" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36969 } } peers { permanent_uuid: "3e10aaab448e49399c885c9e35bbefa7" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 38407 } } peers { permanent_uuid: "bc2a3929a7804f97a09327e5fd735851" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41199 } }
I20260928 14:57:20.058305 8678 ts_tablet_manager.cc:1428] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 14:57:20.058319 8679 raft_consensus.cc:374] T 68ddb4ed8f7a4488b88c682996e0de63 P 3e10aaab448e49399c885c9e35bbefa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:20.058370 8679 raft_consensus.cc:729] T 68ddb4ed8f7a4488b88c682996e0de63 P 3e10aaab448e49399c885c9e35bbefa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e10aaab448e49399c885c9e35bbefa7, State: Initialized, Role: FOLLOWER
I20260928 14:57:20.058450 8664 heartbeater.cc:503] Master 127.20.116.190:40611 was elected leader, sending a full tablet report...
I20260928 14:57:20.058493 8679 consensus_queue.cc:260] T 68ddb4ed8f7a4488b88c682996e0de63 P 3e10aaab448e49399c885c9e35bbefa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9094f10591f4c5cbfc35893eecd660c" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36969 } } peers { permanent_uuid: "3e10aaab448e49399c885c9e35bbefa7" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 38407 } } peers { permanent_uuid: "bc2a3929a7804f97a09327e5fd735851" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41199 } }
I20260928 14:57:20.058808 8533 heartbeater.cc:503] Master 127.20.116.190:40611 was elected leader, sending a full tablet report...
I20260928 14:57:20.059250 8680 raft_consensus.cc:348] T 68ddb4ed8f7a4488b88c682996e0de63 P bc2a3929a7804f97a09327e5fd735851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9094f10591f4c5cbfc35893eecd660c" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36969 } } peers { permanent_uuid: "3e10aaab448e49399c885c9e35bbefa7" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 38407 } } peers { permanent_uuid: "bc2a3929a7804f97a09327e5fd735851" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41199 } }
I20260928 14:57:20.059415 8680 raft_consensus.cc:374] T 68ddb4ed8f7a4488b88c682996e0de63 P bc2a3929a7804f97a09327e5fd735851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:20.059458 8680 raft_consensus.cc:729] T 68ddb4ed8f7a4488b88c682996e0de63 P bc2a3929a7804f97a09327e5fd735851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc2a3929a7804f97a09327e5fd735851, State: Initialized, Role: FOLLOWER
I20260928 14:57:20.058782 8679 ts_tablet_manager.cc:1428] T 68ddb4ed8f7a4488b88c682996e0de63 P 3e10aaab448e49399c885c9e35bbefa7: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 14:57:20.059598 8680 consensus_queue.cc:260] T 68ddb4ed8f7a4488b88c682996e0de63 P bc2a3929a7804f97a09327e5fd735851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9094f10591f4c5cbfc35893eecd660c" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36969 } } peers { permanent_uuid: "3e10aaab448e49399c885c9e35bbefa7" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 38407 } } peers { permanent_uuid: "bc2a3929a7804f97a09327e5fd735851" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41199 } }
I20260928 14:57:20.059897 8402 heartbeater.cc:503] Master 127.20.116.190:40611 was elected leader, sending a full tablet report...
I20260928 14:57:20.060158 8680 ts_tablet_manager.cc:1428] T 68ddb4ed8f7a4488b88c682996e0de63 P bc2a3929a7804f97a09327e5fd735851: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 14:57:20.106680 8684 raft_consensus.cc:482] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:57:20.106812 8684 raft_consensus.cc:504] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9094f10591f4c5cbfc35893eecd660c" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36969 } } peers { permanent_uuid: "3e10aaab448e49399c885c9e35bbefa7" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 38407 } } peers { permanent_uuid: "bc2a3929a7804f97a09327e5fd735851" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41199 } }
I20260928 14:57:20.107216 8684 leader_election.cc:288] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e10aaab448e49399c885c9e35bbefa7 (127.20.116.130:38407), bc2a3929a7804f97a09327e5fd735851 (127.20.116.129:41199)
I20260928 14:57:20.110311 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:20.107560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:47011 (local address 127.20.116.130:38407)
0928 14:57:20.107615 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:20.107618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:20.107629 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:57:20.107732 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:20.107733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:20.107746 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:20.107808 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:20.107971 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.108585 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.109589 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.109966 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.110048 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:20.110058 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:20.110195 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 14:57:20.110233 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 14:57:20.110354 8676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:20.107637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.116.131:53813 (local address 127.20.116.129:41199)
0928 14:57:20.107685 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:20.107687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:20.107851 (+ 164us) server_negotiation.cc:407] Connection header received
0928 14:57:20.107924 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:20.107925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:20.107937 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:20.107994 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:20.108158 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.108626 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.109601 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.109974 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.110056 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:20.110064 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:20.110242 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 14:57:20.110282 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 14:57:20.110364 8666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:20.107450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.130:38407 (local address 127.20.116.131:47011)
0928 14:57:20.107524 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:20.107536 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:20.107598 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:20.107828 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:20.107829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:20.107831 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:20.107904 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:20.107905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.108676 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.108677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:20.109497 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:20.109499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.110049 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.110050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:20.110117 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:20.110119 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:57:20.110189 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:57:20.110233 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 14:57:20.110594 8687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:20.107581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:41199 (local address 127.20.116.131:53813)
0928 14:57:20.107724 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:20.107734 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:20.107863 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:20.107997 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:20.107998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:20.107999 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:20.108071 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:20.108072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.108704 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.108706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:20.109507 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:20.109510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.110056 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.110057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:20.110151 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:20.110154 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:57:20.110232 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:57:20.110272 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260928 14:57:20.110844 8487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ddb4ed8f7a4488b88c682996e0de63" candidate_uuid: "b9094f10591f4c5cbfc35893eecd660c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e10aaab448e49399c885c9e35bbefa7" is_pre_election: true
I20260928 14:57:20.111014 8487 raft_consensus.cc:2454] T 68ddb4ed8f7a4488b88c682996e0de63 P 3e10aaab448e49399c885c9e35bbefa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9094f10591f4c5cbfc35893eecd660c in term 0.
I20260928 14:57:20.111075 8356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ddb4ed8f7a4488b88c682996e0de63" candidate_uuid: "b9094f10591f4c5cbfc35893eecd660c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc2a3929a7804f97a09327e5fd735851" is_pre_election: true
I20260928 14:57:20.111264 8356 raft_consensus.cc:2454] T 68ddb4ed8f7a4488b88c682996e0de63 P bc2a3929a7804f97a09327e5fd735851 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9094f10591f4c5cbfc35893eecd660c in term 0.
I20260928 14:57:20.111248 8553 leader_election.cc:302] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e10aaab448e49399c885c9e35bbefa7, b9094f10591f4c5cbfc35893eecd660c; no voters:
I20260928 14:57:20.111377 8684 raft_consensus.cc:2781] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:57:20.111431 8684 raft_consensus.cc:482] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:57:20.111454 8684 raft_consensus.cc:3037] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:20.112142 8684 raft_consensus.cc:504] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9094f10591f4c5cbfc35893eecd660c" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36969 } } peers { permanent_uuid: "3e10aaab448e49399c885c9e35bbefa7" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 38407 } } peers { permanent_uuid: "bc2a3929a7804f97a09327e5fd735851" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41199 } }
I20260928 14:57:20.112269 8684 leader_election.cc:288] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c [CANDIDATE]: Term 1 election: Requested vote from peers 3e10aaab448e49399c885c9e35bbefa7 (127.20.116.130:38407), bc2a3929a7804f97a09327e5fd735851 (127.20.116.129:41199)
I20260928 14:57:20.112447 8487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ddb4ed8f7a4488b88c682996e0de63" candidate_uuid: "b9094f10591f4c5cbfc35893eecd660c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e10aaab448e49399c885c9e35bbefa7"
I20260928 14:57:20.112493 8356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ddb4ed8f7a4488b88c682996e0de63" candidate_uuid: "b9094f10591f4c5cbfc35893eecd660c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc2a3929a7804f97a09327e5fd735851"
I20260928 14:57:20.112540 8487 raft_consensus.cc:3037] T 68ddb4ed8f7a4488b88c682996e0de63 P 3e10aaab448e49399c885c9e35bbefa7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:20.112572 8356 raft_consensus.cc:3037] T 68ddb4ed8f7a4488b88c682996e0de63 P bc2a3929a7804f97a09327e5fd735851 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:20.113257 8487 raft_consensus.cc:2454] T 68ddb4ed8f7a4488b88c682996e0de63 P 3e10aaab448e49399c885c9e35bbefa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9094f10591f4c5cbfc35893eecd660c in term 1.
I20260928 14:57:20.113401 8356 raft_consensus.cc:2454] T 68ddb4ed8f7a4488b88c682996e0de63 P bc2a3929a7804f97a09327e5fd735851 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9094f10591f4c5cbfc35893eecd660c in term 1.
I20260928 14:57:20.113442 8553 leader_election.cc:302] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e10aaab448e49399c885c9e35bbefa7, b9094f10591f4c5cbfc35893eecd660c; no voters:
I20260928 14:57:20.113567 8684 raft_consensus.cc:2781] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:20.113726 8684 raft_consensus.cc:686] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c [term 1 LEADER]: Becoming Leader. State: Replica: b9094f10591f4c5cbfc35893eecd660c, State: Running, Role: LEADER
I20260928 14:57:20.113826 8684 consensus_queue.cc:237] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9094f10591f4c5cbfc35893eecd660c" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36969 } } peers { permanent_uuid: "3e10aaab448e49399c885c9e35bbefa7" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 38407 } } peers { permanent_uuid: "bc2a3929a7804f97a09327e5fd735851" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41199 } }
I20260928 14:57:20.114728 8214 catalog_manager.cc:5495] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c reported cstate change: term changed from 0 to 1, leader changed from <none> to b9094f10591f4c5cbfc35893eecd660c (127.20.116.131). New cstate: current_term: 1 leader_uuid: "b9094f10591f4c5cbfc35893eecd660c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9094f10591f4c5cbfc35893eecd660c" member_type: VOTER last_known_addr { host: "127.20.116.131" port: 36969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e10aaab448e49399c885c9e35bbefa7" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 38407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc2a3929a7804f97a09327e5fd735851" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41199 } health_report { overall_health: UNKNOWN } } }
I20260928 14:57:20.123167 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:20.120890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47620 (local address 127.20.116.131:36969)
0928 14:57:20.120945 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:20.120947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:20.120960 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:57:20.120994 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:20.120994 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:20.121010 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:20.121067 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:20.121192 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.121643 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.122398 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:20.122614 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:20.122666 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:20.122832 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:20.122925 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:20.122983 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:20.123061 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 14:57:20.123078 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 14:57:20.124979 8356 raft_consensus.cc:1261] T 68ddb4ed8f7a4488b88c682996e0de63 P bc2a3929a7804f97a09327e5fd735851 [term 1 FOLLOWER]: Refusing update from remote peer b9094f10591f4c5cbfc35893eecd660c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:20.124982 8487 raft_consensus.cc:1261] T 68ddb4ed8f7a4488b88c682996e0de63 P 3e10aaab448e49399c885c9e35bbefa7 [term 1 FOLLOWER]: Refusing update from remote peer b9094f10591f4c5cbfc35893eecd660c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:20.125334 8688 consensus_queue.cc:1035] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc2a3929a7804f97a09327e5fd735851" member_type: VOTER last_known_addr { host: "127.20.116.129" port: 41199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:20.125465 8684 consensus_queue.cc:1035] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e10aaab448e49399c885c9e35bbefa7" member_type: VOTER last_known_addr { host: "127.20.116.130" port: 38407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:20.127779 8691 mvcc.cc:205] Tried to move back new op lower bound from 7334328074749214720 to 7334328074707025920. Current Snapshot: MvccSnapshot[applied={T|T < 7334328074749214720}]
I20260928 14:57:20.129429 8692 mvcc.cc:205] Tried to move back new op lower bound from 7334328074749214720 to 7334328074707025920. Current Snapshot: MvccSnapshot[applied={T|T < 7334328074749214720 or (T in {7334328074749214720})}]
I20260928 14:57:20.130133 8213 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:20.130239 8213 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:57:20.131040 8213 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2552a1fafdb34b5fb59e58c86ed969f4: Sending DeleteTablet for 3 replicas of tablet 68ddb4ed8f7a4488b88c682996e0de63
I20260928 14:57:20.131376 8598 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ddb4ed8f7a4488b88c682996e0de63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:20 UTC) from {username='slave'} at 127.0.0.1:47614
I20260928 14:57:20.131451 8336 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ddb4ed8f7a4488b88c682996e0de63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:20 UTC) from {username='slave'} at 127.0.0.1:43360
I20260928 14:57:20.131618 8700 tablet_replica.cc:330] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c: stopping tablet replica
I20260928 14:57:20.131613 8467 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ddb4ed8f7a4488b88c682996e0de63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:57:20 UTC) from {username='slave'} at 127.0.0.1:41826
I20260928 14:57:20.131731 8700 raft_consensus.cc:2229] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:57:20.131827 8700 raft_consensus.cc:2258] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:20.131886 8702 tablet_replica.cc:330] T 68ddb4ed8f7a4488b88c682996e0de63 P 3e10aaab448e49399c885c9e35bbefa7: stopping tablet replica
I20260928 14:57:20.131984 8702 raft_consensus.cc:2229] T 68ddb4ed8f7a4488b88c682996e0de63 P 3e10aaab448e49399c885c9e35bbefa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:57:20.132091 8702 raft_consensus.cc:2258] T 68ddb4ed8f7a4488b88c682996e0de63 P 3e10aaab448e49399c885c9e35bbefa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:57:20.132189 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 8274
I20260928 14:57:20.132525 8702 ts_tablet_manager.cc:1905] T 68ddb4ed8f7a4488b88c682996e0de63 P 3e10aaab448e49399c885c9e35bbefa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:57:20.134179 8702 ts_tablet_manager.cc:1918] T 68ddb4ed8f7a4488b88c682996e0de63 P 3e10aaab448e49399c885c9e35bbefa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:20.134261 8702 log.cc:1196] T 68ddb4ed8f7a4488b88c682996e0de63 P 3e10aaab448e49399c885c9e35bbefa7: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-1/wal/wals/68ddb4ed8f7a4488b88c682996e0de63
I20260928 14:57:20.134500 8702 ts_tablet_manager.cc:1939] T 68ddb4ed8f7a4488b88c682996e0de63 P 3e10aaab448e49399c885c9e35bbefa7: Deleting consensus metadata
I20260928 14:57:20.134912 8200 catalog_manager.cc:4841] TS 3e10aaab448e49399c885c9e35bbefa7 (127.20.116.130:38407): tablet 68ddb4ed8f7a4488b88c682996e0de63 (table test-table [id=de87734593a74375a1641696bc217438]) successfully deleted
I20260928 14:57:20.139034 8700 ts_tablet_manager.cc:1905] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 14:57:20.140727 8200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.116.129:41199 (error 108)
I20260928 14:57:20.140957 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 8405
I20260928 14:57:20.141588 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:20.141090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.129:41199 (local address 127.0.0.1:43366)
0928 14:57:20.141485 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:20.141536 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.129:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369}
I20260928 14:57:20.140974 8700 ts_tablet_manager.cc:1918] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:57:20.141742 8700 log.cc:1196] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c: Deleting WAL directory at /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607192794321-20946-0/minicluster-data/ts-2/wal/wals/68ddb4ed8f7a4488b88c682996e0de63
I20260928 14:57:20.142024 8700 ts_tablet_manager.cc:1939] T 68ddb4ed8f7a4488b88c682996e0de63 P b9094f10591f4c5cbfc35893eecd660c: Deleting consensus metadata
W20260928 14:57:20.142254 8200 catalog_manager.cc:4584] TS bc2a3929a7804f97a09327e5fd735851 (127.20.116.129:41199): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 68ddb4ed8f7a4488b88c682996e0de63: Network error: Client connection negotiation failed: client connection to 127.20.116.129:41199: connect: Connection refused (error 111)
I20260928 14:57:20.142455 8200 catalog_manager.cc:4841] TS b9094f10591f4c5cbfc35893eecd660c (127.20.116.131:36969): tablet 68ddb4ed8f7a4488b88c682996e0de63 (table test-table [id=de87734593a74375a1641696bc217438]) successfully deleted
I20260928 14:57:20.146364 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 8536
I20260928 14:57:20.151340 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 8183
2026-09-28T14:57:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (701 ms)
[----------] 6 tests from EncryptionPolicyTest (8759 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:20 dist-test-slave-pxl6 krb5kdc[8706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:20 dist-test-slave-pxl6 krb5kdc[8706](info): set up 2 sockets
Sep 28 14:57:20 dist-test-slave-pxl6 krb5kdc[8706](info): commencing operation
krb5kdc: starting...
W20260928 14:57:21.199605 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:21 dist-test-slave-pxl6 krb5kdc[8706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:57:21.326879 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37927
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:21.409632 8719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:21.409834 8719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:21.409857 8719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:21.411029 8719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:21.411069 8719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:21.411083 8719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:21.411103 8719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:21.411114 8719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:21.412657 8719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37927
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:21.412914 8719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:21.413154 8719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:21.417979 8724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:21.417994 8727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:21.417994 8725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:21.418210 8719 server_base.cc:1034] running on GCE node
I20260928 14:57:21.418514 8719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:21.418874 8719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:21.420024 8719 hybrid_clock.cc:648] HybridClock initialized: now 1790607441420006 us; error 35 us; skew 500 ppm
Sep 28 14:57:21 dist-test-slave-pxl6 krb5kdc[8706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:21.422694 8719 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:57:21.423185 8719 webserver.cc:460] Webserver started at http://127.0.0.1:36179/ using document root <none> and password file <none>
I20260928 14:57:21.423440 8719 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:21.423485 8719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:21.423609 8719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:21.424687 8719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "23b84baf2b0746a28e0b6a494bd48568"
format_stamp: "Formatted at 2026-09-28 14:57:21 on dist-test-slave-pxl6"
server_key: "7a038551217c0053695bca67e0a7edde"
server_key_iv: "71931d9f3d7c2342a3e4f78b9d063b1e"
server_key_version: "encryptionkey@0"
I20260928 14:57:21.425040 8719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "23b84baf2b0746a28e0b6a494bd48568"
format_stamp: "Formatted at 2026-09-28 14:57:21 on dist-test-slave-pxl6"
server_key: "7a038551217c0053695bca67e0a7edde"
server_key_iv: "71931d9f3d7c2342a3e4f78b9d063b1e"
server_key_version: "encryptionkey@0"
I20260928 14:57:21.426419 8719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 14:57:21.427230 8734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:21.427439 8719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:57:21.427528 8719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "23b84baf2b0746a28e0b6a494bd48568"
format_stamp: "Formatted at 2026-09-28 14:57:21 on dist-test-slave-pxl6"
server_key: "7a038551217c0053695bca67e0a7edde"
server_key_iv: "71931d9f3d7c2342a3e4f78b9d063b1e"
server_key_version: "encryptionkey@0"
I20260928 14:57:21.427603 8719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:21.445312 8719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:21.447445 8719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:21.447589 8719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:21.451818 8719 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37927
I20260928 14:57:21.451875 8786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37927 every 8 connection(s)
I20260928 14:57:21.452275 8719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:21.453186 8787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:21.455706 8787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568: Bootstrap starting.
I20260928 14:57:21.456337 8787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:21.456641 8787 log.cc:824] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:21.457409 8787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568: No bootstrap required, opened a new log
I20260928 14:57:21.459175 8787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b84baf2b0746a28e0b6a494bd48568" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37927 } }
I20260928 14:57:21.459332 8787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:21.459357 8787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b84baf2b0746a28e0b6a494bd48568, State: Initialized, Role: FOLLOWER
I20260928 14:57:21.459514 8787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b84baf2b0746a28e0b6a494bd48568" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37927 } }
I20260928 14:57:21.459591 8787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:21.459615 8787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:21.459663 8787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:21.460222 8787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b84baf2b0746a28e0b6a494bd48568" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37927 } }
I20260928 14:57:21.460351 8787 leader_election.cc:302] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23b84baf2b0746a28e0b6a494bd48568; no voters:
I20260928 14:57:21.460546 8787 leader_election.cc:288] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:21.460642 8790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:21.460851 8787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:21.460850 8790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568 [term 1 LEADER]: Becoming Leader. State: Replica: 23b84baf2b0746a28e0b6a494bd48568, State: Running, Role: LEADER
I20260928 14:57:21.461001 8790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b84baf2b0746a28e0b6a494bd48568" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37927 } }
I20260928 14:57:21.461568 8791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23b84baf2b0746a28e0b6a494bd48568" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b84baf2b0746a28e0b6a494bd48568" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37927 } } }
I20260928 14:57:21.461658 8791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:21.461849 8792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23b84baf2b0746a28e0b6a494bd48568. Latest consensus state: current_term: 1 leader_uuid: "23b84baf2b0746a28e0b6a494bd48568" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b84baf2b0746a28e0b6a494bd48568" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37927 } } }
I20260928 14:57:21.461932 8792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:21.462056 8799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:21.462212 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 8719
I20260928 14:57:21.462327 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:21.462586 20946 external_mini_cluster.cc:1428] Setting key 5029af7b0b562a794371e04dca8dc7f4
I20260928 14:57:21.462666 8799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:21.464506 8799 catalog_manager.cc:1347] Generated new cluster ID: f7d529bae2da4f04ba30b5622a8ac6a8
I20260928 14:57:21.464558 8799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:57:21 dist-test-slave-pxl6 krb5kdc[8706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 14:57:21.477941 8799 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:21.478673 8799 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:21.480199 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:21.463126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34791 (local address 127.0.0.1:37927)
0928 14:57:21.463309 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:21.463318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:21.463350 (+ 32us) server_negotiation.cc:407] Connection header received
0928 14:57:21.463474 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:21.463480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:21.463538 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:21.463690 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:21.470383 (+ 6693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:21.471882 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:21.472969 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:21.473644 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:21.473759 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:21.476621 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:21.476625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:21.476625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:21.476638 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:21.478565 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:21.479002 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:21.479005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:21.479005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:21.479110 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:21.479335 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:21.479336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:21.479336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:21.479574 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:21.479755 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:21.479844 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 14:57:21.479950 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
I20260928 14:57:21.481114 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:57:21.485397 8799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23b84baf2b0746a28e0b6a494bd48568: Generated new TSK 0
I20260928 14:57:21.485603 8799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:21.489482 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:21.482510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34437 (local address 127.0.0.1:37927)
0928 14:57:21.482576 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:21.482578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:21.482700 (+ 122us) server_negotiation.cc:407] Connection header received
0928 14:57:21.482773 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:21.482774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:21.482784 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:21.482829 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:21.483364 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:21.484213 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:21.484960 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:21.485285 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:21.485372 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:21.486752 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:21.486753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:21.486754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:21.486761 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:21.488287 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:21.488680 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:21.488681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:21.488681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:21.488738 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:21.488963 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:21.488964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:21.488964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:21.489075 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:21.489208 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:21.489313 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 14:57:21.489374 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:57:21.491448 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 8719
W20260928 14:57:21.496261 8704 connection.cc:535] client connection to 127.0.0.1:37927 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37927): Connection reset by peer (error 104)
2026-09-28T14:57:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:21 dist-test-slave-pxl6 krb5kdc[8818](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:21 dist-test-slave-pxl6 krb5kdc[8818](info): set up 2 sockets
Sep 28 14:57:21 dist-test-slave-pxl6 krb5kdc[8818](info): commencing operation
krb5kdc: starting...
W20260928 14:57:22.531471 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:22 dist-test-slave-pxl6 krb5kdc[8818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:57:22.654009 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44983
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:22.738729 8831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:22.738997 8831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:22.739042 8831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:22.740336 8831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:22.740422 8831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:22.740445 8831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:22.740466 8831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:22.740499 8831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:22.742132 8831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44983
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:22.742445 8831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:22.742708 8831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:22.747876 8836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:22.747882 8839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:22.747895 8837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:22.748167 8831 server_base.cc:1034] running on GCE node
I20260928 14:57:22.748399 8831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:22.748646 8831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:22.749830 8831 hybrid_clock.cc:648] HybridClock initialized: now 1790607442749808 us; error 36 us; skew 500 ppm
Sep 28 14:57:22 dist-test-slave-pxl6 krb5kdc[8818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:22.758036 8831 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:57:22.758515 8831 webserver.cc:460] Webserver started at http://127.0.0.1:37629/ using document root <none> and password file <none>
I20260928 14:57:22.758775 8831 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:22.758826 8831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:22.758962 8831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:22.760039 8831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "1ab35f4d15094a0f88a8f0175c741fba"
format_stamp: "Formatted at 2026-09-28 14:57:22 on dist-test-slave-pxl6"
server_key: "d5ee816bdd1dd75ca021690b2ad3688b"
server_key_iv: "2f664f90941eecd74943d4a8ef78106d"
server_key_version: "encryptionkey@0"
I20260928 14:57:22.760401 8831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "1ab35f4d15094a0f88a8f0175c741fba"
format_stamp: "Formatted at 2026-09-28 14:57:22 on dist-test-slave-pxl6"
server_key: "d5ee816bdd1dd75ca021690b2ad3688b"
server_key_iv: "2f664f90941eecd74943d4a8ef78106d"
server_key_version: "encryptionkey@0"
I20260928 14:57:22.761821 8831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:57:22.762629 8846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:22.762801 8831 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:57:22.762887 8831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "1ab35f4d15094a0f88a8f0175c741fba"
format_stamp: "Formatted at 2026-09-28 14:57:22 on dist-test-slave-pxl6"
server_key: "d5ee816bdd1dd75ca021690b2ad3688b"
server_key_iv: "2f664f90941eecd74943d4a8ef78106d"
server_key_version: "encryptionkey@0"
I20260928 14:57:22.762943 8831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:22.775313 8831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:22.777585 8831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:22.777711 8831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:22.781821 8831 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44983
I20260928 14:57:22.781854 8898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44983 every 8 connection(s)
I20260928 14:57:22.782229 8831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:22.783241 8899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:22.785633 8899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba: Bootstrap starting.
I20260928 14:57:22.786264 8899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:22.786594 8899 log.cc:824] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:22.787361 8899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba: No bootstrap required, opened a new log
I20260928 14:57:22.789023 8899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab35f4d15094a0f88a8f0175c741fba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44983 } }
I20260928 14:57:22.789192 8899 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:22.789242 8899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ab35f4d15094a0f88a8f0175c741fba, State: Initialized, Role: FOLLOWER
I20260928 14:57:22.789408 8899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab35f4d15094a0f88a8f0175c741fba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44983 } }
I20260928 14:57:22.789494 8899 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:22.789503 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 8831
I20260928 14:57:22.789539 8899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:22.789593 8899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:22.789629 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:22.789822 20946 external_mini_cluster.cc:1428] Setting key ffc4ab41f737fd768a0b432100f942a1
I20260928 14:57:22.790391 8899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab35f4d15094a0f88a8f0175c741fba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44983 } }
I20260928 14:57:22.790510 8899 leader_election.cc:302] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ab35f4d15094a0f88a8f0175c741fba; no voters:
I20260928 14:57:22.790721 8899 leader_election.cc:288] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:22.790755 8904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:22.790920 8904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba [term 1 LEADER]: Becoming Leader. State: Replica: 1ab35f4d15094a0f88a8f0175c741fba, State: Running, Role: LEADER
I20260928 14:57:22.790984 8899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:22.791054 8904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab35f4d15094a0f88a8f0175c741fba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44983 } }
I20260928 14:57:22.791662 8905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ab35f4d15094a0f88a8f0175c741fba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab35f4d15094a0f88a8f0175c741fba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44983 } } }
I20260928 14:57:22.791781 8905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:22.792140 8913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:22.792150 8906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ab35f4d15094a0f88a8f0175c741fba. Latest consensus state: current_term: 1 leader_uuid: "1ab35f4d15094a0f88a8f0175c741fba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab35f4d15094a0f88a8f0175c741fba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44983 } } }
I20260928 14:57:22.792316 8906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:22.792819 8913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:22.794476 8913 catalog_manager.cc:1347] Generated new cluster ID: cd89290e1d7341efbd946dcaf0a9f70b
I20260928 14:57:22.794556 8913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:22.796110 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:22.790340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47541 (local address 127.0.0.1:44983)
0928 14:57:22.790632 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:22.790644 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:22.790689 (+ 45us) server_negotiation.cc:407] Connection header received
0928 14:57:22.790844 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:22.790852 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:22.790915 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:22.791344 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:22.791444 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.792992 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.793983 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.794611 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.794715 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:22.795471 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:22.795474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:22.795475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:22.795654 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:22.795684 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:22.795695 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:22.795797 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:22.795870 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 14:57:22.795900 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":229,"thread_start_us":113,"threads_started":1}
I20260928 14:57:22.797024 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:57:22.801574 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:22.798485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49793 (local address 127.0.0.1:44983)
0928 14:57:22.798522 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:22.798524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:22.798553 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:57:22.798619 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:22.798620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:22.798630 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:22.798685 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:22.798832 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.799598 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.800318 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:22.800558 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:22.800640 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:22.801319 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:22.801320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:22.801320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:22.801325 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:22.801338 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:22.801343 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:22.801396 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:22.801466 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 14:57:22.801479 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:57:22.814569 8913 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:22.815135 8913 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:22.822410 8913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ab35f4d15094a0f88a8f0175c741fba: Generated new TSK 0
I20260928 14:57:22.822661 8913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:22.831068 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 8831
W20260928 14:57:22.835361 8816 connection.cc:535] client connection to 127.0.0.1:44983 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44983): Connection reset by peer (error 104)
2026-09-28T14:57:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:22 dist-test-slave-pxl6 krb5kdc[8930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:22 dist-test-slave-pxl6 krb5kdc[8930](info): set up 2 sockets
Sep 28 14:57:22 dist-test-slave-pxl6 krb5kdc[8930](info): commencing operation
krb5kdc: starting...
W20260928 14:57:23.867635 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:23 dist-test-slave-pxl6 krb5kdc[8930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:57:23.994206 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33619
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:24.077147 8943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:24.077423 8943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:24.077471 8943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:24.078715 8943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:24.078788 8943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:24.078812 8943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:24.078833 8943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:24.078900 8943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:24.080473 8943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33619
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:24.080754 8943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:24.080988 8943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:57:24.086378 8943 server_base.cc:1034] running on GCE node
W20260928 14:57:24.086337 8948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:24.086304 8949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:24.086324 8951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:24.086886 8943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:24.087245 8943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:24.088449 8943 hybrid_clock.cc:648] HybridClock initialized: now 1790607444088426 us; error 42 us; skew 500 ppm
Sep 28 14:57:24 dist-test-slave-pxl6 krb5kdc[8930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:24.091272 8943 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:57:24.091749 8943 webserver.cc:460] Webserver started at http://127.0.0.1:44023/ using document root <none> and password file <none>
I20260928 14:57:24.091944 8943 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:24.091986 8943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:24.092085 8943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:24.093110 8943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "89ea5cd682de49beb7e7a91e004533d3"
format_stamp: "Formatted at 2026-09-28 14:57:24 on dist-test-slave-pxl6"
server_key: "f4900a57b32547b31a68afe8a80f8b1f"
server_key_iv: "9d07724de5e8315669fe736e5e6ff086"
server_key_version: "encryptionkey@0"
I20260928 14:57:24.093420 8943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "89ea5cd682de49beb7e7a91e004533d3"
format_stamp: "Formatted at 2026-09-28 14:57:24 on dist-test-slave-pxl6"
server_key: "f4900a57b32547b31a68afe8a80f8b1f"
server_key_iv: "9d07724de5e8315669fe736e5e6ff086"
server_key_version: "encryptionkey@0"
I20260928 14:57:24.094832 8943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:57:24.095583 8958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:24.095813 8943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:57:24.095919 8943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "89ea5cd682de49beb7e7a91e004533d3"
format_stamp: "Formatted at 2026-09-28 14:57:24 on dist-test-slave-pxl6"
server_key: "f4900a57b32547b31a68afe8a80f8b1f"
server_key_iv: "9d07724de5e8315669fe736e5e6ff086"
server_key_version: "encryptionkey@0"
I20260928 14:57:24.095992 8943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:24.099407 8943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:24.101454 8943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:24.101614 8943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:24.105965 8943 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33619
I20260928 14:57:24.105993 9010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33619 every 8 connection(s)
I20260928 14:57:24.106345 8943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:24.107261 9011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:24.109481 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 8943
I20260928 14:57:24.109604 9011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3: Bootstrap starting.
I20260928 14:57:24.109676 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:24.109877 20946 external_mini_cluster.cc:1428] Setting key deba207d990f6d99304285c28225a135
I20260928 14:57:24.110289 9011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:24.110699 9011 log.cc:824] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:24.111497 9011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3: No bootstrap required, opened a new log
I20260928 14:57:24.112669 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:24.110417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58381 (local address 127.0.0.1:33619)
0928 14:57:24.110651 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:24.110659 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:24.110685 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:57:24.111182 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:24.111189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:24.111216 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:24.111337 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:24.112192 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:24.112195 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:24.112196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:24.112200 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:24.112238 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:24.112248 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:24.112321 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:24.112389 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 14:57:24.112446 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":94,"threads_started":1}
I20260928 14:57:24.114007 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:57:24.114050 9011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ea5cd682de49beb7e7a91e004533d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } }
I20260928 14:57:24.114229 9011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:24.114279 9011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89ea5cd682de49beb7e7a91e004533d3, State: Initialized, Role: FOLLOWER
I20260928 14:57:24.114403 9011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ea5cd682de49beb7e7a91e004533d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } }
I20260928 14:57:24.114490 9011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:24.114528 9011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:24.114611 9011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:24.115223 9011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ea5cd682de49beb7e7a91e004533d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } }
I20260928 14:57:24.115339 9011 leader_election.cc:302] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89ea5cd682de49beb7e7a91e004533d3; no voters:
I20260928 14:57:24.115536 9011 leader_election.cc:288] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:24.115581 9020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:24.115844 9020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3 [term 1 LEADER]: Becoming Leader. State: Replica: 89ea5cd682de49beb7e7a91e004533d3, State: Running, Role: LEADER
I20260928 14:57:24.115866 9011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:24.115952 9020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ea5cd682de49beb7e7a91e004533d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } }
I20260928 14:57:24.116374 9023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89ea5cd682de49beb7e7a91e004533d3. Latest consensus state: current_term: 1 leader_uuid: "89ea5cd682de49beb7e7a91e004533d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ea5cd682de49beb7e7a91e004533d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } } }
I20260928 14:57:24.116484 9023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:24.116724 9022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89ea5cd682de49beb7e7a91e004533d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ea5cd682de49beb7e7a91e004533d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } } }
I20260928 14:57:24.116798 9022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3 [sys.catalog]: This master's current role is: LEADER
W20260928 14:57:24.117691 9037 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:57:24.117792 9037 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:57:24.117899 9030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:24.118114 9030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:24.119191 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:24.115653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33621 (local address 127.0.0.1:33619)
0928 14:57:24.115714 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:24.115716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:24.117468 (+ 1752us) server_negotiation.cc:407] Connection header received
0928 14:57:24.117583 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:24.117584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:24.117591 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:24.117638 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:24.118934 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:24.118935 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:24.118936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:24.118941 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:24.118959 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:24.118963 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:24.119046 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:24.119112 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 14:57:24.119126 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 14:57:24.119941 9030 catalog_manager.cc:1347] Generated new cluster ID: 742a0cc22ca041438b6c49259055e589
I20260928 14:57:24.119995 9030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:24.167289 9030 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:24.167516 9030 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:24.173542 9030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89ea5cd682de49beb7e7a91e004533d3: Generated new TSK 0
I20260928 14:57:24.173763 9030 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:24.177443 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 8943
2026-09-28T14:57:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1347 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:24 dist-test-slave-pxl6 krb5kdc[9042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:24 dist-test-slave-pxl6 krb5kdc[9042](info): set up 2 sockets
Sep 28 14:57:24 dist-test-slave-pxl6 krb5kdc[9042](info): commencing operation
krb5kdc: starting...
W20260928 14:57:25.219792 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:25 dist-test-slave-pxl6 krb5kdc[9042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:57:25.342989 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36347
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:25.426931 9055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:25.427184 9055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:25.427222 9055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:25.428364 9055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:25.428442 9055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:25.428465 9055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:25.428486 9055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:25.428515 9055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:25.430014 9055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:25.430305 9055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:25.430547 9055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:25.435453 9063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:25.435474 9060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:25.435549 9061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:25.435768 9055 server_base.cc:1034] running on GCE node
I20260928 14:57:25.436030 9055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:25.436388 9055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:25.437638 9055 hybrid_clock.cc:648] HybridClock initialized: now 1790607445437606 us; error 39 us; skew 500 ppm
Sep 28 14:57:25 dist-test-slave-pxl6 krb5kdc[9042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:25.440354 9055 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:57:25.440874 9055 webserver.cc:460] Webserver started at http://127.0.0.1:46207/ using document root <none> and password file <none>
I20260928 14:57:25.441133 9055 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:25.441206 9055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:25.441368 9055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:25.442595 9055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "3822d259ab3d4893977ec0f44abe1665"
format_stamp: "Formatted at 2026-09-28 14:57:25 on dist-test-slave-pxl6"
server_key: "fd881d3327cf47997e6e5a4d76e6408c"
server_key_iv: "52e5b331c319a817948629766be6955f"
server_key_version: "encryptionkey@0"
I20260928 14:57:25.443003 9055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "3822d259ab3d4893977ec0f44abe1665"
format_stamp: "Formatted at 2026-09-28 14:57:25 on dist-test-slave-pxl6"
server_key: "fd881d3327cf47997e6e5a4d76e6408c"
server_key_iv: "52e5b331c319a817948629766be6955f"
server_key_version: "encryptionkey@0"
I20260928 14:57:25.444468 9055 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:57:25.445242 9070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:25.445415 9055 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:25.445521 9055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "3822d259ab3d4893977ec0f44abe1665"
format_stamp: "Formatted at 2026-09-28 14:57:25 on dist-test-slave-pxl6"
server_key: "fd881d3327cf47997e6e5a4d76e6408c"
server_key_iv: "52e5b331c319a817948629766be6955f"
server_key_version: "encryptionkey@0"
I20260928 14:57:25.445596 9055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:25.460028 9055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:25.461983 9055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:25.462152 9055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:25.466246 9055 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36347
I20260928 14:57:25.466286 9122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36347 every 8 connection(s)
I20260928 14:57:25.466701 9055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:25.467574 9123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:25.468199 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 9055
I20260928 14:57:25.468315 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:25.468519 20946 external_mini_cluster.cc:1428] Setting key d7a237190de56db3544470675ccc6aa6
I20260928 14:57:25.470878 9123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665: Bootstrap starting.
I20260928 14:57:25.471475 9123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:25.471799 9123 log.cc:824] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:25.472556 9123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665: No bootstrap required, opened a new log
I20260928 14:57:25.475199 9123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3822d259ab3d4893977ec0f44abe1665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36347 } }
I20260928 14:57:25.475340 9123 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:25.475363 9123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3822d259ab3d4893977ec0f44abe1665, State: Initialized, Role: FOLLOWER
I20260928 14:57:25.475495 9123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3822d259ab3d4893977ec0f44abe1665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36347 } }
I20260928 14:57:25.475564 9123 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 28 14:57:25 dist-test-slave-pxl6 krb5kdc[9042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 14:57:25.475587 9123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:25.475654 9123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:25.476286 9123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3822d259ab3d4893977ec0f44abe1665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36347 } }
I20260928 14:57:25.476415 9123 leader_election.cc:302] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3822d259ab3d4893977ec0f44abe1665; no voters:
I20260928 14:57:25.476588 9123 leader_election.cc:288] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:25.476624 9128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:25.476763 9128 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665 [term 1 LEADER]: Becoming Leader. State: Replica: 3822d259ab3d4893977ec0f44abe1665, State: Running, Role: LEADER
I20260928 14:57:25.476867 9123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:25.476883 9128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3822d259ab3d4893977ec0f44abe1665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36347 } }
I20260928 14:57:25.477190 9128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3822d259ab3d4893977ec0f44abe1665. Latest consensus state: current_term: 1 leader_uuid: "3822d259ab3d4893977ec0f44abe1665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3822d259ab3d4893977ec0f44abe1665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36347 } } }
I20260928 14:57:25.477283 9128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:25.477458 9130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3822d259ab3d4893977ec0f44abe1665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3822d259ab3d4893977ec0f44abe1665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36347 } } }
I20260928 14:57:25.477509 9134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:25.477615 9130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:25.478199 9134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:25.480057 9126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:25.469098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38607 (local address 127.0.0.1:36347)
0928 14:57:25.469266 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:25.469274 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:25.469301 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:57:25.469708 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:25.469714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:25.469761 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:25.469874 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:25.470751 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:25.471737 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:25.472739 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:25.473133 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:25.473219 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:25.476074 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:25.476079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:25.476079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:25.476092 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:25.478289 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:25.478821 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:25.478823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:25.478824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:25.478906 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:25.479149 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:25.479150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:25.479150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:25.479469 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:25.479639 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:25.479742 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 14:57:25.479887 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260928 14:57:25.480177 9134 catalog_manager.cc:1347] Generated new cluster ID: 69b1d0c0ace14be294401172f3d6bc94
I20260928 14:57:25.480237 9134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:25.480978 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:57:25.489629 9134 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:25.490334 9134 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:25.493005 9126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:25.482977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45893 (local address 127.0.0.1:36347)
0928 14:57:25.483013 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:25.483015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:25.486690 (+ 3675us) server_negotiation.cc:407] Connection header received
0928 14:57:25.486773 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:25.486775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:25.486786 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:25.486859 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:25.487402 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:25.488045 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:25.488755 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:25.488979 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:25.489051 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:25.490335 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:25.490336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:25.490337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:25.490344 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:25.491836 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:25.492205 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:25.492207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:25.492207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:25.492276 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:25.492476 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:25.492477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:25.492477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:25.492591 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:25.492712 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:25.492809 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 14:57:25.492898 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":17}
I20260928 14:57:25.501132 9134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3822d259ab3d4893977ec0f44abe1665: Generated new TSK 0
I20260928 14:57:25.501364 9134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:25.504531 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 9055
2026-09-28T14:57:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:25 dist-test-slave-pxl6 krb5kdc[9154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:25 dist-test-slave-pxl6 krb5kdc[9154](info): set up 2 sockets
Sep 28 14:57:25 dist-test-slave-pxl6 krb5kdc[9154](info): commencing operation
krb5kdc: starting...
W20260928 14:57:26.543434 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:26 dist-test-slave-pxl6 krb5kdc[9154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:57:26.669399 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37833
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:26.752595 9167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:26.752808 9167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:26.752833 9167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:26.753942 9167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:26.753978 9167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:26.753991 9167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:26.754004 9167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:26.754016 9167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:26.755441 9167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:26.755640 9167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:26.755827 9167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:26.760841 9172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:26.760948 9167 server_base.cc:1034] running on GCE node
W20260928 14:57:26.760804 9173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:26.760804 9175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:26.761377 9167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:26.761679 9167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:26.762847 9167 hybrid_clock.cc:648] HybridClock initialized: now 1790607446762844 us; error 36 us; skew 500 ppm
Sep 28 14:57:26 dist-test-slave-pxl6 krb5kdc[9154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:26.765298 9167 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:57:26.765785 9167 webserver.cc:460] Webserver started at http://127.0.0.1:34513/ using document root <none> and password file <none>
I20260928 14:57:26.766019 9167 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:26.766088 9167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:26.766244 9167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:26.767400 9167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "8ea2de48365f4be2904143f1432dd8da"
format_stamp: "Formatted at 2026-09-28 14:57:26 on dist-test-slave-pxl6"
server_key: "f3e5f09228ec8573e22b4be282d3f06c"
server_key_iv: "92cdeb06e39020587db2575c90170d86"
server_key_version: "encryptionkey@0"
I20260928 14:57:26.767845 9167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "8ea2de48365f4be2904143f1432dd8da"
format_stamp: "Formatted at 2026-09-28 14:57:26 on dist-test-slave-pxl6"
server_key: "f3e5f09228ec8573e22b4be282d3f06c"
server_key_iv: "92cdeb06e39020587db2575c90170d86"
server_key_version: "encryptionkey@0"
I20260928 14:57:26.769279 9167 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:57:26.770056 9182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:26.770373 9167 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:57:26.770493 9167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "8ea2de48365f4be2904143f1432dd8da"
format_stamp: "Formatted at 2026-09-28 14:57:26 on dist-test-slave-pxl6"
server_key: "f3e5f09228ec8573e22b4be282d3f06c"
server_key_iv: "92cdeb06e39020587db2575c90170d86"
server_key_version: "encryptionkey@0"
I20260928 14:57:26.770627 9167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:26.783871 9167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:26.786165 9167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:26.786346 9167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:26.790728 9167 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37833
I20260928 14:57:26.790776 9234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37833 every 8 connection(s)
I20260928 14:57:26.791141 9167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:26.792032 9235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:26.794461 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 9167
I20260928 14:57:26.794607 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:26.794590 9235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da: Bootstrap starting.
I20260928 14:57:26.794849 20946 external_mini_cluster.cc:1428] Setting key d9cfdab802c6af59c80161c8a8f9da46
I20260928 14:57:26.795385 9235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:26.795799 9235 log.cc:824] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:26.796610 9235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da: No bootstrap required, opened a new log
I20260928 14:57:26.799047 9235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea2de48365f4be2904143f1432dd8da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37833 } }
I20260928 14:57:26.799232 9235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:26.799263 9235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea2de48365f4be2904143f1432dd8da, State: Initialized, Role: FOLLOWER
I20260928 14:57:26.799394 9235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea2de48365f4be2904143f1432dd8da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37833 } }
I20260928 14:57:26.799466 9235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:26.799495 9235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:26.799540 9235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:26.800324 9235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea2de48365f4be2904143f1432dd8da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37833 } }
I20260928 14:57:26.800436 9235 leader_election.cc:302] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ea2de48365f4be2904143f1432dd8da; no voters:
I20260928 14:57:26.800633 9235 leader_election.cc:288] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:26.800668 9240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:26.800841 9240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da [term 1 LEADER]: Becoming Leader. State: Replica: 8ea2de48365f4be2904143f1432dd8da, State: Running, Role: LEADER
I20260928 14:57:26.800935 9235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:26.800976 9240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea2de48365f4be2904143f1432dd8da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37833 } }
I20260928 14:57:26.801448 9242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ea2de48365f4be2904143f1432dd8da. Latest consensus state: current_term: 1 leader_uuid: "8ea2de48365f4be2904143f1432dd8da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea2de48365f4be2904143f1432dd8da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37833 } } }
I20260928 14:57:26.801527 9242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:26.801537 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:26.795418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43265 (local address 127.0.0.1:37833)
0928 14:57:26.795635 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:26.795646 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:26.795686 (+ 40us) server_negotiation.cc:407] Connection header received
0928 14:57:26.796319 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:26.796329 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:26.796395 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:26.796551 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:26.796806 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:26.798196 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:26.799050 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:26.799661 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:26.799795 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:26.800542 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:26.800546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:26.800547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:26.800783 (+ 236us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:26.800823 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:26.800838 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:26.801199 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:26.801237 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 14:57:26.801311 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":92,"threads_started":1}
I20260928 14:57:26.802093 9241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ea2de48365f4be2904143f1432dd8da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea2de48365f4be2904143f1432dd8da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37833 } } }
I20260928 14:57:26.802181 9241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:26.802757 9249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:26.802969 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:57:26.803313 9249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:26.805151 9249 catalog_manager.cc:1347] Generated new cluster ID: 0b6cbb18ecdc4845879910ddaad23165
I20260928 14:57:26.805238 9249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:26.808120 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:26.804522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52677 (local address 127.0.0.1:37833)
0928 14:57:26.804560 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:26.804562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:26.804683 (+ 121us) server_negotiation.cc:407] Connection header received
0928 14:57:26.804738 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:26.804739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:26.804751 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:26.804814 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:26.804953 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:26.805646 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:26.806749 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:26.807072 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:26.807151 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:26.807863 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:26.807865 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:26.807865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:26.807870 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:26.807888 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:26.807893 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:26.807956 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:26.808010 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 14:57:26.808036 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:57:26.810563 9249 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:26.811288 9249 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:26.818998 9249 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ea2de48365f4be2904143f1432dd8da: Generated new TSK 0
I20260928 14:57:26.819231 9249 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:26.828804 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 9167
2026-09-28T14:57:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:26 dist-test-slave-pxl6 krb5kdc[9267](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:26 dist-test-slave-pxl6 krb5kdc[9267](info): set up 2 sockets
Sep 28 14:57:26 dist-test-slave-pxl6 krb5kdc[9267](info): commencing operation
krb5kdc: starting...
W20260928 14:57:27.943499 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:28 dist-test-slave-pxl6 krb5kdc[9267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:57:28.074421 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39661
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:28.157770 9280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:28.158001 9280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:28.158028 9280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:28.159353 9280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:28.159407 9280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:28.159422 9280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:28.159435 9280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:28.159449 9280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:28.161053 9280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:28.161345 9280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:28.161593 9280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:28.166875 9288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:28.166867 9285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:28.166858 9286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:28.167130 9280 server_base.cc:1034] running on GCE node
I20260928 14:57:28.167409 9280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:28.167696 9280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:28.168867 9280 hybrid_clock.cc:648] HybridClock initialized: now 1790607448168854 us; error 36 us; skew 500 ppm
Sep 28 14:57:28 dist-test-slave-pxl6 krb5kdc[9267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:28.177120 9280 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:57:28.177644 9280 webserver.cc:460] Webserver started at http://127.0.0.1:37469/ using document root <none> and password file <none>
I20260928 14:57:28.177904 9280 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:28.177953 9280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:28.178130 9280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:28.179308 9280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "47dec3cf70b542108bbe21e7011ad28c"
format_stamp: "Formatted at 2026-09-28 14:57:28 on dist-test-slave-pxl6"
server_key: "899345384dd9b1d713be30542e5f47fc"
server_key_iv: "fdf8ba84133c8dfb8c0db5c72f9ce537"
server_key_version: "encryptionkey@0"
I20260928 14:57:28.179677 9280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "47dec3cf70b542108bbe21e7011ad28c"
format_stamp: "Formatted at 2026-09-28 14:57:28 on dist-test-slave-pxl6"
server_key: "899345384dd9b1d713be30542e5f47fc"
server_key_iv: "fdf8ba84133c8dfb8c0db5c72f9ce537"
server_key_version: "encryptionkey@0"
I20260928 14:57:28.181109 9280 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:57:28.181953 9295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:28.182156 9280 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:57:28.182258 9280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "47dec3cf70b542108bbe21e7011ad28c"
format_stamp: "Formatted at 2026-09-28 14:57:28 on dist-test-slave-pxl6"
server_key: "899345384dd9b1d713be30542e5f47fc"
server_key_iv: "fdf8ba84133c8dfb8c0db5c72f9ce537"
server_key_version: "encryptionkey@0"
I20260928 14:57:28.182336 9280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:28.186101 9280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:28.188634 9280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:28.188750 9280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:28.192915 9280 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39661
I20260928 14:57:28.192945 9347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39661 every 8 connection(s)
I20260928 14:57:28.193307 9280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:28.194286 9348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:28.196698 9348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c: Bootstrap starting.
I20260928 14:57:28.197302 9348 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:28.197609 9348 log.cc:824] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:28.198354 9348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c: No bootstrap required, opened a new log
I20260928 14:57:28.199631 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 9280
I20260928 14:57:28.199725 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:28.199937 20946 external_mini_cluster.cc:1428] Setting key a3b96f1267f39bfd39941a7e04756dd6
I20260928 14:57:28.200242 9348 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47dec3cf70b542108bbe21e7011ad28c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39661 } }
I20260928 14:57:28.200541 9348 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:28.200662 9348 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47dec3cf70b542108bbe21e7011ad28c, State: Initialized, Role: FOLLOWER
I20260928 14:57:28.200845 9348 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47dec3cf70b542108bbe21e7011ad28c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39661 } }
I20260928 14:57:28.200927 9348 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:28.200951 9348 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:28.201023 9348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:28.201820 9348 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47dec3cf70b542108bbe21e7011ad28c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39661 } }
I20260928 14:57:28.201961 9348 leader_election.cc:302] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47dec3cf70b542108bbe21e7011ad28c; no voters:
I20260928 14:57:28.202144 9348 leader_election.cc:288] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:28.202186 9353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:28.202387 9353 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c [term 1 LEADER]: Becoming Leader. State: Replica: 47dec3cf70b542108bbe21e7011ad28c, State: Running, Role: LEADER
I20260928 14:57:28.202451 9352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:28.200508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44677 (local address 127.0.0.1:39661)
0928 14:57:28.200697 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:28.200705 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:28.200752 (+ 47us) server_negotiation.cc:407] Connection header received
0928 14:57:28.200904 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:28.200909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:28.200954 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:28.201123 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:28.201886 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:28.201908 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:28.201908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:28.201915 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:28.201962 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:28.201984 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:28.202070 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:28.202164 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 14:57:28.202210 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260928 14:57:28.202483 9348 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:28.202771 9353 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47dec3cf70b542108bbe21e7011ad28c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39661 } }
I20260928 14:57:28.203472 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:57:28.203671 9354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47dec3cf70b542108bbe21e7011ad28c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47dec3cf70b542108bbe21e7011ad28c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39661 } } }
I20260928 14:57:28.203778 9354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:28.204007 9355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47dec3cf70b542108bbe21e7011ad28c. Latest consensus state: current_term: 1 leader_uuid: "47dec3cf70b542108bbe21e7011ad28c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47dec3cf70b542108bbe21e7011ad28c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39661 } } }
I20260928 14:57:28.204276 9355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c [sys.catalog]: This master's current role is: LEADER
W20260928 14:57:28.204358 9367 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:57:28.204417 9367 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:57:28.204670 9368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:28.204864 9368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:28.206679 9368 catalog_manager.cc:1347] Generated new cluster ID: 8e0183e7f5284d81955f2773234ea13b
I20260928 14:57:28.206732 9368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:28.207248 9352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:28.205872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40345 (local address 127.0.0.1:39661)
0928 14:57:28.205911 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:28.205913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:28.206083 (+ 170us) server_negotiation.cc:407] Connection header received
0928 14:57:28.206155 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:28.206156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:28.206162 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:28.206223 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:28.206985 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:28.206986 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:28.206987 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:28.206992 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:28.207007 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:28.207012 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:28.207043 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:28.207167 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 14:57:28.207181 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:57:28.220283 9368 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:28.220482 9368 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:28.234112 9368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47dec3cf70b542108bbe21e7011ad28c: Generated new TSK 0
I20260928 14:57:28.234351 9368 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:28.236981 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 9280
2026-09-28T14:57:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1408 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:28 dist-test-slave-pxl6 krb5kdc[9379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:28 dist-test-slave-pxl6 krb5kdc[9379](info): set up 2 sockets
Sep 28 14:57:28 dist-test-slave-pxl6 krb5kdc[9379](info): commencing operation
krb5kdc: starting...
W20260928 14:57:29.279775 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:29 dist-test-slave-pxl6 krb5kdc[9379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:57:29.406842 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39681
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:29.491626 9392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:29.491889 9392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:29.491933 9392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:29.493271 9392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:29.493333 9392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:29.493357 9392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:29.493381 9392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:29.493407 9392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:29.495033 9392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:29.495342 9392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:29.495585 9392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:29.501151 9398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:29.501205 9400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:29.501266 9392 server_base.cc:1034] running on GCE node
W20260928 14:57:29.501212 9397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:29.501644 9392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:29.501919 9392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:29.503090 9392 hybrid_clock.cc:648] HybridClock initialized: now 1790607449503080 us; error 34 us; skew 500 ppm
Sep 28 14:57:29 dist-test-slave-pxl6 krb5kdc[9379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:29.505690 9392 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:57:29.506152 9392 webserver.cc:460] Webserver started at http://127.0.0.1:34577/ using document root <none> and password file <none>
I20260928 14:57:29.506434 9392 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:29.506501 9392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:29.506706 9392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:29.507831 9392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "b2c4fe505ab14214a381f0d14965de2f"
format_stamp: "Formatted at 2026-09-28 14:57:29 on dist-test-slave-pxl6"
server_key: "5150ef9c4cbdc6919c19dd576372b4bc"
server_key_iv: "1d1838e5c3da06e5b964c219abf740d5"
server_key_version: "encryptionkey@0"
I20260928 14:57:29.508219 9392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "b2c4fe505ab14214a381f0d14965de2f"
format_stamp: "Formatted at 2026-09-28 14:57:29 on dist-test-slave-pxl6"
server_key: "5150ef9c4cbdc6919c19dd576372b4bc"
server_key_iv: "1d1838e5c3da06e5b964c219abf740d5"
server_key_version: "encryptionkey@0"
I20260928 14:57:29.509804 9392 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:29.510891 9407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:29.511085 9392 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:57:29.511214 9392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "b2c4fe505ab14214a381f0d14965de2f"
format_stamp: "Formatted at 2026-09-28 14:57:29 on dist-test-slave-pxl6"
server_key: "5150ef9c4cbdc6919c19dd576372b4bc"
server_key_iv: "1d1838e5c3da06e5b964c219abf740d5"
server_key_version: "encryptionkey@0"
I20260928 14:57:29.511299 9392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:29.525571 9392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:29.527694 9392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:29.527870 9392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:29.532143 9392 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39681
I20260928 14:57:29.532181 9459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39681 every 8 connection(s)
I20260928 14:57:29.532557 9392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:29.533497 9460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:29.535908 9460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f: Bootstrap starting.
I20260928 14:57:29.536518 9460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:29.536830 9460 log.cc:824] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:29.537534 9460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f: No bootstrap required, opened a new log
I20260928 14:57:29.539181 9460 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c4fe505ab14214a381f0d14965de2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39681 } }
I20260928 14:57:29.539326 9460 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:29.539350 9460 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2c4fe505ab14214a381f0d14965de2f, State: Initialized, Role: FOLLOWER
I20260928 14:57:29.539517 9460 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c4fe505ab14214a381f0d14965de2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39681 } }
I20260928 14:57:29.539582 9460 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:29.539602 9460 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:29.539670 9460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:29.540203 9460 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c4fe505ab14214a381f0d14965de2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39681 } }
I20260928 14:57:29.540323 9460 leader_election.cc:302] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2c4fe505ab14214a381f0d14965de2f; no voters:
I20260928 14:57:29.540505 9460 leader_election.cc:288] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:29.540550 9463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:29.540772 9460 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:29.540803 9463 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f [term 1 LEADER]: Becoming Leader. State: Replica: b2c4fe505ab14214a381f0d14965de2f, State: Running, Role: LEADER
I20260928 14:57:29.540966 9463 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c4fe505ab14214a381f0d14965de2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39681 } }
I20260928 14:57:29.541388 9465 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2c4fe505ab14214a381f0d14965de2f. Latest consensus state: current_term: 1 leader_uuid: "b2c4fe505ab14214a381f0d14965de2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c4fe505ab14214a381f0d14965de2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39681 } } }
I20260928 14:57:29.541491 9465 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:29.541519 9464 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2c4fe505ab14214a381f0d14965de2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c4fe505ab14214a381f0d14965de2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39681 } } }
I20260928 14:57:29.541599 9464 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:29.541733 9472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:29.542264 9472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:29.542624 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 9392
I20260928 14:57:29.542760 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:29.543000 20946 external_mini_cluster.cc:1428] Setting key 7b7ac5b66697ecbbb633f77d49589e96
I20260928 14:57:29.544050 9472 catalog_manager.cc:1347] Generated new cluster ID: aa6ebbbd61154c8fac87bf67ef0fb7da
I20260928 14:57:29.544103 9472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:57:29 dist-test-slave-pxl6 krb5kdc[9379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 14:57:29.551571 9472 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:29.552291 9472 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:29.553913 9482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:29.543433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:34441 (local address 127.0.0.1:39681)
0928 14:57:29.543731 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:29.543741 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:29.543945 (+ 204us) server_negotiation.cc:407] Connection header received
0928 14:57:29.544165 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:29.544173 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:29.544249 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:29.544383 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:29.545032 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:29.545951 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:29.546782 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:29.547264 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:29.547390 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:29.549785 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:29.549789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:29.549789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:29.549805 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:29.551861 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:29.552309 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:29.552313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:29.552314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:29.552421 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:29.552672 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:29.552673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:29.552673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:29.552955 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:29.553098 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:29.553564 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 14:57:29.553665 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":74,"threads_started":1}
I20260928 14:57:29.554795 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:57:29.571008 9472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2c4fe505ab14214a381f0d14965de2f: Generated new TSK 0
I20260928 14:57:29.571252 9472 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:29.571509 9482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:29.562923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:58511 (local address 127.0.0.1:39681)
0928 14:57:29.562977 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:29.562980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:29.563187 (+ 207us) server_negotiation.cc:407] Connection header received
0928 14:57:29.563259 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:29.563260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:29.563272 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:29.563319 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:29.564027 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:29.565181 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:29.566300 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:29.566632 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:29.566695 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:29.568229 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:29.568231 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:29.568231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:29.568239 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:29.569875 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:29.570325 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:29.570326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:29.570326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:29.570391 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:29.570884 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:29.570885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:29.570886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:29.571025 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:29.571109 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:29.571361 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 14:57:29.571432 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 14:57:29.573465 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 9392
W20260928 14:57:29.577817 9377 connection.cc:535] client connection to 127.0.0.1:39681 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39681): Connection reset by peer (error 104)
2026-09-28T14:57:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:29 dist-test-slave-pxl6 krb5kdc[9491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:29 dist-test-slave-pxl6 krb5kdc[9491](info): set up 2 sockets
Sep 28 14:57:29 dist-test-slave-pxl6 krb5kdc[9491](info): commencing operation
krb5kdc: starting...
W20260928 14:57:30.620000 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:30 dist-test-slave-pxl6 krb5kdc[9491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:57:30.746251 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34193
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:30.829517 9504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:30.829733 9504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:30.829756 9504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:30.830891 9504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:30.830928 9504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:30.830942 9504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:30.830955 9504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:30.830966 9504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:30.832394 9504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:30.832598 9504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:30.832772 9504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:30.837774 9512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:30.837761 9509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:30.837757 9510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:30.838039 9504 server_base.cc:1034] running on GCE node
I20260928 14:57:30.838356 9504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:30.838737 9504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:30.839887 9504 hybrid_clock.cc:648] HybridClock initialized: now 1790607450839875 us; error 27 us; skew 500 ppm
Sep 28 14:57:30 dist-test-slave-pxl6 krb5kdc[9491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:30.842409 9504 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:57:30.842955 9504 webserver.cc:460] Webserver started at http://127.0.0.1:41295/ using document root <none> and password file <none>
I20260928 14:57:30.843194 9504 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:30.843261 9504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:30.843421 9504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:30.844498 9504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "053a57c3463b4f7abc12acaa5cc5b9fc"
format_stamp: "Formatted at 2026-09-28 14:57:30 on dist-test-slave-pxl6"
server_key: "c7258d952303313423db5ace3f5328d5"
server_key_iv: "b568b04b3132f441a13f608cd1b419da"
server_key_version: "encryptionkey@0"
I20260928 14:57:30.844868 9504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "053a57c3463b4f7abc12acaa5cc5b9fc"
format_stamp: "Formatted at 2026-09-28 14:57:30 on dist-test-slave-pxl6"
server_key: "c7258d952303313423db5ace3f5328d5"
server_key_iv: "b568b04b3132f441a13f608cd1b419da"
server_key_version: "encryptionkey@0"
I20260928 14:57:30.846249 9504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:57:30.847083 9519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:30.847280 9504 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:30.847383 9504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "053a57c3463b4f7abc12acaa5cc5b9fc"
format_stamp: "Formatted at 2026-09-28 14:57:30 on dist-test-slave-pxl6"
server_key: "c7258d952303313423db5ace3f5328d5"
server_key_iv: "b568b04b3132f441a13f608cd1b419da"
server_key_version: "encryptionkey@0"
I20260928 14:57:30.847471 9504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:30.859783 9504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:30.861552 9504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:30.861719 9504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:30.866185 9504 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34193
I20260928 14:57:30.866210 9571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34193 every 8 connection(s)
I20260928 14:57:30.866631 9504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:30.867481 9572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:30.869761 9572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc: Bootstrap starting.
I20260928 14:57:30.870357 9572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:30.870666 9572 log.cc:824] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:30.871234 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 9504
I20260928 14:57:30.871366 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:30.871445 9572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc: No bootstrap required, opened a new log
I20260928 14:57:30.871606 20946 external_mini_cluster.cc:1428] Setting key ed0fa7bf09291b1e09f170e4157902ff
I20260928 14:57:30.873596 9572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053a57c3463b4f7abc12acaa5cc5b9fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34193 } }
I20260928 14:57:30.873777 9572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:30.873867 9572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 053a57c3463b4f7abc12acaa5cc5b9fc, State: Initialized, Role: FOLLOWER
I20260928 14:57:30.874017 9572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053a57c3463b4f7abc12acaa5cc5b9fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34193 } }
I20260928 14:57:30.874092 9572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:30.874122 9572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:30.874166 9572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:30.874909 9572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053a57c3463b4f7abc12acaa5cc5b9fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34193 } }
I20260928 14:57:30.875015 9572 leader_election.cc:302] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 053a57c3463b4f7abc12acaa5cc5b9fc; no voters:
I20260928 14:57:30.875157 9572 leader_election.cc:288] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:30.875216 9577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:30.875394 9572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:30.875411 9577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc [term 1 LEADER]: Becoming Leader. State: Replica: 053a57c3463b4f7abc12acaa5cc5b9fc, State: Running, Role: LEADER
I20260928 14:57:30.875586 9577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053a57c3463b4f7abc12acaa5cc5b9fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34193 } }
I20260928 14:57:30.875981 9578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "053a57c3463b4f7abc12acaa5cc5b9fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053a57c3463b4f7abc12acaa5cc5b9fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34193 } } }
I20260928 14:57:30.876065 9578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:30.876264 9579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 053a57c3463b4f7abc12acaa5cc5b9fc. Latest consensus state: current_term: 1 leader_uuid: "053a57c3463b4f7abc12acaa5cc5b9fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053a57c3463b4f7abc12acaa5cc5b9fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34193 } } }
I20260928 14:57:30.876557 9579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:30.876791 9586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:30.877502 9586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:30.878827 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:30.872195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:35245 (local address 127.0.0.1:34193)
0928 14:57:30.872393 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:30.872403 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:30.872437 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:57:30.873145 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:30.873154 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:30.873232 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:30.873407 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:30.873597 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:30.875372 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:30.876683 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:30.877258 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:30.877406 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:30.878151 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:30.878155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:30.878156 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:30.878339 (+ 183us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:30.878373 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:30.878383 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:30.878479 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:30.878584 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 14:57:30.878612 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":72,"threads_started":1}
I20260928 14:57:30.879180 9586 catalog_manager.cc:1347] Generated new cluster ID: 49f7e4f1f4f64a93b6837fc2ff39448a
I20260928 14:57:30.879266 9586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:30.879779 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:57:30.885864 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:30.882878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:38367 (local address 127.0.0.1:34193)
0928 14:57:30.882919 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:30.882920 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:30.883063 (+ 143us) server_negotiation.cc:407] Connection header received
0928 14:57:30.883141 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:30.883142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:30.883153 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:30.883221 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:30.883379 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:30.883861 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:30.884597 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:30.884817 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:30.884888 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:30.885577 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:30.885579 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:30.885579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:30.885583 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:30.885598 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:30.885602 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:30.885679 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:30.885743 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 14:57:30.885752 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:57:30.889107 9586 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:30.889842 9586 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:30.894448 9586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 053a57c3463b4f7abc12acaa5cc5b9fc: Generated new TSK 0
I20260928 14:57:30.894695 9586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:30.902346 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 9504
W20260928 14:57:30.906919 9489 connection.cc:535] client connection to 127.0.0.1:34193 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34193): Connection reset by peer (error 104)
2026-09-28T14:57:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:30 dist-test-slave-pxl6 krb5kdc[9603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:30 dist-test-slave-pxl6 krb5kdc[9603](info): set up 2 sockets
Sep 28 14:57:30 dist-test-slave-pxl6 krb5kdc[9603](info): commencing operation
krb5kdc: starting...
W20260928 14:57:31.959661 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:32 dist-test-slave-pxl6 krb5kdc[9603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:57:32.086217 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38393
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:32.168701 9616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:32.168952 9616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:32.168990 9616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:32.170153 9616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:32.170226 9616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:32.170250 9616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:32.170279 9616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:32.170308 9616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:32.171823 9616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:32.172122 9616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:32.172379 9616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:32.177321 9624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:32.177313 9621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:32.177313 9622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:32.177803 9616 server_base.cc:1034] running on GCE node
I20260928 14:57:32.178090 9616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:32.178395 9616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:32.179555 9616 hybrid_clock.cc:648] HybridClock initialized: now 1790607452179500 us; error 80 us; skew 500 ppm
Sep 28 14:57:32 dist-test-slave-pxl6 krb5kdc[9603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:32.182675 9616 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:57:32.183324 9616 webserver.cc:460] Webserver started at http://127.0.0.1:41265/ using document root <none> and password file <none>
I20260928 14:57:32.183591 9616 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:32.183662 9616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:32.183859 9616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:32.184968 9616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "f089f56c92d74ae7bc12867fb7155901"
format_stamp: "Formatted at 2026-09-28 14:57:32 on dist-test-slave-pxl6"
server_key: "ac72eb39873c62bc56e27c4666187fbd"
server_key_iv: "9e89f32cd7118453c8ab046e1badcfb4"
server_key_version: "encryptionkey@0"
I20260928 14:57:32.185348 9616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "f089f56c92d74ae7bc12867fb7155901"
format_stamp: "Formatted at 2026-09-28 14:57:32 on dist-test-slave-pxl6"
server_key: "ac72eb39873c62bc56e27c4666187fbd"
server_key_iv: "9e89f32cd7118453c8ab046e1badcfb4"
server_key_version: "encryptionkey@0"
I20260928 14:57:32.186872 9616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:32.187638 9631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:32.187812 9616 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:57:32.187919 9616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "f089f56c92d74ae7bc12867fb7155901"
format_stamp: "Formatted at 2026-09-28 14:57:32 on dist-test-slave-pxl6"
server_key: "ac72eb39873c62bc56e27c4666187fbd"
server_key_iv: "9e89f32cd7118453c8ab046e1badcfb4"
server_key_version: "encryptionkey@0"
I20260928 14:57:32.187994 9616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:32.191486 9616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:32.193303 9616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:32.193459 9616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:32.197382 9616 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38393
I20260928 14:57:32.197428 9683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38393 every 8 connection(s)
I20260928 14:57:32.197784 9616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:32.198750 9684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:32.201092 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 9616
I20260928 14:57:32.201149 9684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901: Bootstrap starting.
I20260928 14:57:32.201246 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:32.201444 20946 external_mini_cluster.cc:1428] Setting key 8658c113ad1648967cc8566c4c325597
I20260928 14:57:32.201895 9684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:32.202281 9684 log.cc:824] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:32.203117 9684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901: No bootstrap required, opened a new log
I20260928 14:57:32.204375 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:32.201973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:51241 (local address 127.0.0.1:38393)
0928 14:57:32.202204 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:32.202215 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:32.202253 (+ 38us) server_negotiation.cc:407] Connection header received
0928 14:57:32.202913 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:32.202923 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:32.202960 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:32.203117 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:32.203913 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:32.203918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:32.203919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:32.203924 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:32.203970 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:32.203984 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:32.204085 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:32.204114 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 14:57:32.204159 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":98,"threads_started":1}
I20260928 14:57:32.205536 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:57:32.205693 9684 raft_consensus.cc:348] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f089f56c92d74ae7bc12867fb7155901" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38393 } }
I20260928 14:57:32.205870 9684 raft_consensus.cc:374] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:32.205900 9684 raft_consensus.cc:729] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f089f56c92d74ae7bc12867fb7155901, State: Initialized, Role: FOLLOWER
I20260928 14:57:32.206027 9684 consensus_queue.cc:260] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f089f56c92d74ae7bc12867fb7155901" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38393 } }
I20260928 14:57:32.206130 9684 raft_consensus.cc:388] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:32.206162 9684 raft_consensus.cc:482] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:32.206209 9684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:32.207067 9684 raft_consensus.cc:504] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f089f56c92d74ae7bc12867fb7155901" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38393 } }
I20260928 14:57:32.207198 9684 leader_election.cc:302] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f089f56c92d74ae7bc12867fb7155901; no voters:
I20260928 14:57:32.207376 9684 leader_election.cc:288] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:32.207407 9694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:32.207670 9694 raft_consensus.cc:686] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901 [term 1 LEADER]: Becoming Leader. State: Replica: f089f56c92d74ae7bc12867fb7155901, State: Running, Role: LEADER
I20260928 14:57:32.207706 9684 sys_catalog.cc:564] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:32.207861 9694 consensus_queue.cc:237] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f089f56c92d74ae7bc12867fb7155901" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38393 } }
I20260928 14:57:32.208238 9694 sys_catalog.cc:455] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f089f56c92d74ae7bc12867fb7155901. Latest consensus state: current_term: 1 leader_uuid: "f089f56c92d74ae7bc12867fb7155901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f089f56c92d74ae7bc12867fb7155901" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38393 } } }
I20260928 14:57:32.208253 9695 sys_catalog.cc:455] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f089f56c92d74ae7bc12867fb7155901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f089f56c92d74ae7bc12867fb7155901" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38393 } } }
I20260928 14:57:32.208372 9694 sys_catalog.cc:458] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:32.208441 9695 sys_catalog.cc:458] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:32.208541 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:32.207165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:37821 (local address 127.0.0.1:38393)
0928 14:57:32.207208 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:32.207210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:32.207349 (+ 139us) server_negotiation.cc:407] Connection header received
0928 14:57:32.207424 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:32.207425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:32.207431 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:32.207495 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:32.208167 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:32.208168 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:32.208168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:32.208172 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:32.208187 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:32.208191 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:32.208252 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:32.208380 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 14:57:32.208397 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:57:32.208695 9703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:32.209290 9703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:32.210893 9703 catalog_manager.cc:1347] Generated new cluster ID: bf6c4e270aca40998fa1fe8a4824deaa
I20260928 14:57:32.210963 9703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:32.219892 9703 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:32.220081 9703 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:32.225708 9703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f089f56c92d74ae7bc12867fb7155901: Generated new TSK 0
I20260928 14:57:32.225912 9703 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:32.231985 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 9616
2026-09-28T14:57:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:32 dist-test-slave-pxl6 krb5kdc[9715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:32 dist-test-slave-pxl6 krb5kdc[9715](info): set up 2 sockets
Sep 28 14:57:32 dist-test-slave-pxl6 krb5kdc[9715](info): commencing operation
krb5kdc: starting...
W20260928 14:57:33.267725 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:33 dist-test-slave-pxl6 krb5kdc[9715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:57:33.389158 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43397
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:33.470656 9728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:33.470901 9728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:33.470935 9728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:33.472065 9728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:33.472131 9728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:33.472154 9728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:33.472177 9728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:33.472231 9728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:33.473702 9728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43397
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:33.473987 9728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:33.474224 9728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:33.479422 9733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:33.479415 9736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:33.479415 9734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:33.479447 9728 server_base.cc:1034] running on GCE node
I20260928 14:57:33.479918 9728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:33.480293 9728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:33.481473 9728 hybrid_clock.cc:648] HybridClock initialized: now 1790607453481450 us; error 38 us; skew 500 ppm
Sep 28 14:57:33 dist-test-slave-pxl6 krb5kdc[9715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:33.489248 9728 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:57:33.489835 9728 webserver.cc:460] Webserver started at http://127.0.0.1:34369/ using document root <none> and password file <none>
I20260928 14:57:33.490118 9728 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:33.490191 9728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:33.490351 9728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:33.491621 9728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "a85f8736fa1f4e0183233038158f4f6c"
format_stamp: "Formatted at 2026-09-28 14:57:33 on dist-test-slave-pxl6"
server_key: "146e84de5f2d5d57e28041ddd223bedd"
server_key_iv: "acbc4b7fcaa737306f6ca078b829e93c"
server_key_version: "encryptionkey@0"
I20260928 14:57:33.492020 9728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "a85f8736fa1f4e0183233038158f4f6c"
format_stamp: "Formatted at 2026-09-28 14:57:33 on dist-test-slave-pxl6"
server_key: "146e84de5f2d5d57e28041ddd223bedd"
server_key_iv: "acbc4b7fcaa737306f6ca078b829e93c"
server_key_version: "encryptionkey@0"
I20260928 14:57:33.493474 9728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:33.494272 9743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:33.494446 9728 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:57:33.494565 9728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "a85f8736fa1f4e0183233038158f4f6c"
format_stamp: "Formatted at 2026-09-28 14:57:33 on dist-test-slave-pxl6"
server_key: "146e84de5f2d5d57e28041ddd223bedd"
server_key_iv: "acbc4b7fcaa737306f6ca078b829e93c"
server_key_version: "encryptionkey@0"
I20260928 14:57:33.494659 9728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:33.535032 9728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:33.537745 9728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:33.537933 9728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:33.542631 9728 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43397
I20260928 14:57:33.542682 9795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43397 every 8 connection(s)
I20260928 14:57:33.543102 9728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:33.543998 9796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:33.544144 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 9728
I20260928 14:57:33.544287 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:33.544513 20946 external_mini_cluster.cc:1428] Setting key 3e44aef47507777dc8aa6bf7f80994f7
I20260928 14:57:33.547643 9796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c: Bootstrap starting.
I20260928 14:57:33.548436 9796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:33.548735 9796 log.cc:824] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:33.549495 9796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c: No bootstrap required, opened a new log
Sep 28 14:57:33 dist-test-slave-pxl6 krb5kdc[9715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 14:57:33.552201 9796 raft_consensus.cc:348] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a85f8736fa1f4e0183233038158f4f6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } }
I20260928 14:57:33.552371 9796 raft_consensus.cc:374] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:33.552418 9796 raft_consensus.cc:729] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a85f8736fa1f4e0183233038158f4f6c, State: Initialized, Role: FOLLOWER
I20260928 14:57:33.552574 9796 consensus_queue.cc:260] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a85f8736fa1f4e0183233038158f4f6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } }
I20260928 14:57:33.552678 9796 raft_consensus.cc:388] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:33.552716 9796 raft_consensus.cc:482] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:33.552765 9796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:33.553447 9796 raft_consensus.cc:504] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a85f8736fa1f4e0183233038158f4f6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } }
I20260928 14:57:33.553560 9796 leader_election.cc:302] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a85f8736fa1f4e0183233038158f4f6c; no voters:
I20260928 14:57:33.553763 9796 leader_election.cc:288] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:33.553818 9801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:33.553993 9801 raft_consensus.cc:686] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c [term 1 LEADER]: Becoming Leader. State: Replica: a85f8736fa1f4e0183233038158f4f6c, State: Running, Role: LEADER
I20260928 14:57:33.554082 9796 sys_catalog.cc:564] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:33.554148 9801 consensus_queue.cc:237] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a85f8736fa1f4e0183233038158f4f6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } }
I20260928 14:57:33.554593 9801 sys_catalog.cc:455] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a85f8736fa1f4e0183233038158f4f6c. Latest consensus state: current_term: 1 leader_uuid: "a85f8736fa1f4e0183233038158f4f6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a85f8736fa1f4e0183233038158f4f6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } } }
I20260928 14:57:33.554875 9801 sys_catalog.cc:458] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:33.554798 9803 sys_catalog.cc:455] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a85f8736fa1f4e0183233038158f4f6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a85f8736fa1f4e0183233038158f4f6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } } }
I20260928 14:57:33.555334 9803 sys_catalog.cc:458] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:33.555495 9810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:33.556484 9799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:33.545144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:39169 (local address 127.0.0.1:43397)
0928 14:57:33.545332 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:33.545340 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:33.545368 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:57:33.545875 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:33.545882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:33.545939 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:33.546064 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:33.546914 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:33.547852 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:33.548928 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:33.549480 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:33.549583 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:33.552309 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:33.552313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:33.552313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:33.552325 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:33.554576 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:33.555119 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:33.555123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:33.555123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:33.555247 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:33.555537 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:33.555538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:33.555539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:33.555890 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:33.556075 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:33.556244 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 14:57:33.556331 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260928 14:57:33.556650 9810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:33.557471 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:57:33.558975 9810 catalog_manager.cc:1347] Generated new cluster ID: 2a15b8648f3b4367b99fca766f2a2bcf
I20260928 14:57:33.559031 9810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:33.573004 9799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:33.562827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:56881 (local address 127.0.0.1:43397)
0928 14:57:33.562873 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:33.562875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:33.566744 (+ 3869us) server_negotiation.cc:407] Connection header received
0928 14:57:33.566820 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:33.566821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:33.566838 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:33.566903 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:33.567482 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:33.567977 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:33.568731 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:33.568974 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:33.569062 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:33.570293 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:33.570295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:33.570295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:33.570303 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:33.571789 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:33.572180 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:33.572181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:33.572182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:33.572243 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:33.572466 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:33.572468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:33.572468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:33.572583 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:33.572703 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:33.572818 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 14:57:33.572883 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:57:33.575626 9810 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:33.576326 9810 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:33.578816 9810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a85f8736fa1f4e0183233038158f4f6c: Generated new TSK 0
I20260928 14:57:33.579008 9810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:33.584113 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 9728
W20260928 14:57:33.589015 9713 connection.cc:535] client connection to 127.0.0.1:43397 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43397): Connection reset by peer (error 104)
2026-09-28T14:57:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:33 dist-test-slave-pxl6 krb5kdc[9827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:33 dist-test-slave-pxl6 krb5kdc[9827](info): set up 2 sockets
Sep 28 14:57:33 dist-test-slave-pxl6 krb5kdc[9827](info): commencing operation
krb5kdc: starting...
W20260928 14:57:34.619702 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:34 dist-test-slave-pxl6 krb5kdc[9827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:57:34.750957 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39193
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:34.837466 9840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:34.837708 9840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:34.837734 9840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:34.839000 9840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:34.839044 9840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:34.839059 9840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:34.839071 9840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:34.839084 9840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:34.840565 9840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39193
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:34.840777 9840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:34.840951 9840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:34.846053 9846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:34.846217 9845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:34.846050 9848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:34.846339 9840 server_base.cc:1034] running on GCE node
I20260928 14:57:34.846626 9840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:34.846943 9840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:34.848125 9840 hybrid_clock.cc:648] HybridClock initialized: now 1790607454848106 us; error 33 us; skew 500 ppm
Sep 28 14:57:34 dist-test-slave-pxl6 krb5kdc[9827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:34.850880 9840 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:57:34.851428 9840 webserver.cc:460] Webserver started at http://127.0.0.1:41587/ using document root <none> and password file <none>
I20260928 14:57:34.851666 9840 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:34.851716 9840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:34.851888 9840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:34.852998 9840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "9b046671f48947498f650a15c48b4551"
format_stamp: "Formatted at 2026-09-28 14:57:34 on dist-test-slave-pxl6"
server_key: "64c90dc1d123356fbd004a8b56a65368"
server_key_iv: "90cf6300184defb5b3b54d048eed7f7e"
server_key_version: "encryptionkey@0"
I20260928 14:57:34.853374 9840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "9b046671f48947498f650a15c48b4551"
format_stamp: "Formatted at 2026-09-28 14:57:34 on dist-test-slave-pxl6"
server_key: "64c90dc1d123356fbd004a8b56a65368"
server_key_iv: "90cf6300184defb5b3b54d048eed7f7e"
server_key_version: "encryptionkey@0"
I20260928 14:57:34.854822 9840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:34.855566 9855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:34.855774 9840 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:57:34.855880 9840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "9b046671f48947498f650a15c48b4551"
format_stamp: "Formatted at 2026-09-28 14:57:34 on dist-test-slave-pxl6"
server_key: "64c90dc1d123356fbd004a8b56a65368"
server_key_iv: "90cf6300184defb5b3b54d048eed7f7e"
server_key_version: "encryptionkey@0"
I20260928 14:57:34.855965 9840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:34.872205 9840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:34.874150 9840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:34.874336 9840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:34.879210 9840 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39193
I20260928 14:57:34.879249 9907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39193 every 8 connection(s)
I20260928 14:57:34.879662 9840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:34.880748 9908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:34.883445 9908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551: Bootstrap starting.
I20260928 14:57:34.884146 9908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:34.884472 9908 log.cc:824] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:34.885264 9908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551: No bootstrap required, opened a new log
I20260928 14:57:34.886363 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 9840
I20260928 14:57:34.886464 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:34.886804 20946 external_mini_cluster.cc:1428] Setting key 4ee327ebfb091f45972a60a17c8c7942
I20260928 14:57:34.887607 9908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b046671f48947498f650a15c48b4551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39193 } }
I20260928 14:57:34.887804 9908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:34.887848 9908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b046671f48947498f650a15c48b4551, State: Initialized, Role: FOLLOWER
I20260928 14:57:34.888062 9908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b046671f48947498f650a15c48b4551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39193 } }
I20260928 14:57:34.888175 9908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:34.888217 9908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:34.888270 9908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:34.888937 9908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b046671f48947498f650a15c48b4551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39193 } }
I20260928 14:57:34.889073 9908 leader_election.cc:302] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b046671f48947498f650a15c48b4551; no voters:
I20260928 14:57:34.889268 9908 leader_election.cc:288] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:34.889298 9913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:34.889472 9913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551 [term 1 LEADER]: Becoming Leader. State: Replica: 9b046671f48947498f650a15c48b4551, State: Running, Role: LEADER
I20260928 14:57:34.889578 9908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:34.889621 9913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b046671f48947498f650a15c48b4551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39193 } }
I20260928 14:57:34.890748 9914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b046671f48947498f650a15c48b4551. Latest consensus state: current_term: 1 leader_uuid: "9b046671f48947498f650a15c48b4551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b046671f48947498f650a15c48b4551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39193 } } }
I20260928 14:57:34.890837 9914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:34.891090 9926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:34.891120 9915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b046671f48947498f650a15c48b4551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b046671f48947498f650a15c48b4551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39193 } } }
I20260928 14:57:34.891281 9915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:34.891808 9926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:34.893304 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:34.887387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:53749 (local address 127.0.0.1:39193)
0928 14:57:34.887574 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:34.887582 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:34.887608 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:57:34.887708 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:34.887713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:34.887758 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:34.887907 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:34.888212 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:34.889746 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:34.891011 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:34.891689 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:34.891804 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:34.892587 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:34.892591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:34.892591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:34.892775 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:34.892811 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:34.892822 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:34.892917 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:34.893024 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 14:57:34.893057 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260928 14:57:34.893954 9926 catalog_manager.cc:1347] Generated new cluster ID: b8fc6858c50244f19c10a950228e7b0b
I20260928 14:57:34.894009 9926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:34.894318 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:57:34.902103 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:34.898904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:42607 (local address 127.0.0.1:39193)
0928 14:57:34.898948 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:34.898950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:34.899119 (+ 169us) server_negotiation.cc:407] Connection header received
0928 14:57:34.899202 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:34.899203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:34.899226 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:34.899302 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:34.899457 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:34.899970 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:34.900727 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:34.900938 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:34.901024 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:34.901733 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:34.901735 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:34.901735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:34.901740 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:34.901756 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:34.901761 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:34.901830 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:34.901976 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 14:57:34.901997 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:57:34.940263 9926 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:34.940850 9926 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:34.948603 9926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b046671f48947498f650a15c48b4551: Generated new TSK 0
I20260928 14:57:34.948822 9926 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:34.958052 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 9840
W20260928 14:57:34.962815 9825 connection.cc:535] client connection to 127.0.0.1:39193 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39193): Connection reset by peer (error 104)
2026-09-28T14:57:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:35 dist-test-slave-pxl6 krb5kdc[9939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:35 dist-test-slave-pxl6 krb5kdc[9939](info): set up 2 sockets
Sep 28 14:57:35 dist-test-slave-pxl6 krb5kdc[9939](info): commencing operation
krb5kdc: starting...
W20260928 14:57:36.003808 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:36 dist-test-slave-pxl6 krb5kdc[9939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:57:36.127372 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34801
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:36.208467 9952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:36.208709 9952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:36.208748 9952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:36.209990 9952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:36.210060 9952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:36.210084 9952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:36.210107 9952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:36.210135 9952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:36.211656 9952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34801
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:36.211953 9952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:36.212190 9952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:36.217190 9960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:36.217372 9952 server_base.cc:1034] running on GCE node
W20260928 14:57:36.217237 9958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:36.217216 9957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:36.217724 9952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:36.218062 9952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:36.219228 9952 hybrid_clock.cc:648] HybridClock initialized: now 1790607456219217 us; error 38 us; skew 500 ppm
Sep 28 14:57:36 dist-test-slave-pxl6 krb5kdc[9939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:36.221764 9952 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:57:36.222256 9952 webserver.cc:460] Webserver started at http://127.0.0.1:44329/ using document root <none> and password file <none>
I20260928 14:57:36.222498 9952 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:36.222589 9952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:36.222739 9952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:36.223841 9952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "27f121d1f1084f84803808fdbb4e66c5"
format_stamp: "Formatted at 2026-09-28 14:57:36 on dist-test-slave-pxl6"
server_key: "4e6cdad3dfffd646dc04698856b333e4"
server_key_iv: "134cf9169976a3d7f3fc171ac48fe226"
server_key_version: "encryptionkey@0"
I20260928 14:57:36.224211 9952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "27f121d1f1084f84803808fdbb4e66c5"
format_stamp: "Formatted at 2026-09-28 14:57:36 on dist-test-slave-pxl6"
server_key: "4e6cdad3dfffd646dc04698856b333e4"
server_key_iv: "134cf9169976a3d7f3fc171ac48fe226"
server_key_version: "encryptionkey@0"
I20260928 14:57:36.225606 9952 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:57:36.226500 9967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:36.226730 9952 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:57:36.226856 9952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "27f121d1f1084f84803808fdbb4e66c5"
format_stamp: "Formatted at 2026-09-28 14:57:36 on dist-test-slave-pxl6"
server_key: "4e6cdad3dfffd646dc04698856b333e4"
server_key_iv: "134cf9169976a3d7f3fc171ac48fe226"
server_key_version: "encryptionkey@0"
I20260928 14:57:36.226943 9952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:36.230261 9952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:36.232318 9952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:36.232461 9952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:36.236323 9952 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34801
I20260928 14:57:36.236378 10019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34801 every 8 connection(s)
I20260928 14:57:36.236702 9952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:36.238010 10020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:36.240351 10020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5: Bootstrap starting.
I20260928 14:57:36.240962 10020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:36.241255 10020 log.cc:824] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:36.242048 10020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5: No bootstrap required, opened a new log
I20260928 14:57:36.242138 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 9952
I20260928 14:57:36.242292 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:36.242506 20946 external_mini_cluster.cc:1428] Setting key 6446f0f9f5d5fc6cf62e43a27c9919ce
I20260928 14:57:36.244266 10020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f121d1f1084f84803808fdbb4e66c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34801 } }
I20260928 14:57:36.244457 10020 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:36.244500 10020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f121d1f1084f84803808fdbb4e66c5, State: Initialized, Role: FOLLOWER
I20260928 14:57:36.244661 10020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f121d1f1084f84803808fdbb4e66c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34801 } }
I20260928 14:57:36.244776 10020 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:36.244808 10020 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:36.244853 10020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:36.245780 10024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:36.243092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:37337 (local address 127.0.0.1:34801)
0928 14:57:36.243309 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:36.243318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:36.243345 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:57:36.244239 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:36.244246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:36.244275 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:36.244409 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:36.245267 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:36.245271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:36.245271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:36.245276 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:36.245322 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:36.245334 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:36.245391 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:36.245493 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 14:57:36.245530 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
I20260928 14:57:36.246157 10020 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f121d1f1084f84803808fdbb4e66c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34801 } }
I20260928 14:57:36.246276 10020 leader_election.cc:302] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27f121d1f1084f84803808fdbb4e66c5; no voters:
I20260928 14:57:36.246443 10020 leader_election.cc:288] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:36.246631 10025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:36.246757 10025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5 [term 1 LEADER]: Becoming Leader. State: Replica: 27f121d1f1084f84803808fdbb4e66c5, State: Running, Role: LEADER
I20260928 14:57:36.246768 10020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:36.246852 10025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f121d1f1084f84803808fdbb4e66c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34801 } }
I20260928 14:57:36.246999 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:57:36.247215 10025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27f121d1f1084f84803808fdbb4e66c5. Latest consensus state: current_term: 1 leader_uuid: "27f121d1f1084f84803808fdbb4e66c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f121d1f1084f84803808fdbb4e66c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34801 } } }
I20260928 14:57:36.247316 10025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:36.247225 10027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27f121d1f1084f84803808fdbb4e66c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f121d1f1084f84803808fdbb4e66c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34801 } } }
I20260928 14:57:36.247459 10027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:36.247542 10033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:36.248096 10033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:36.249933 10024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:36.248659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:34269 (local address 127.0.0.1:34801)
0928 14:57:36.248694 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:36.248696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:36.248838 (+ 142us) server_negotiation.cc:407] Connection header received
0928 14:57:36.248909 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:36.248910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:36.248914 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:36.248987 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:36.249700 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:36.249701 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:36.249701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:36.249705 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:36.249720 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:36.249724 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:36.249776 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:36.249849 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 14:57:36.249865 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 14:57:36.251902 10033 catalog_manager.cc:1347] Generated new cluster ID: a339b23e0383402a8c47790e24ccda1c
I20260928 14:57:36.251958 10033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:36.257330 10033 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:36.257500 10033 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:36.272414 10033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27f121d1f1084f84803808fdbb4e66c5: Generated new TSK 0
I20260928 14:57:36.272631 10033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:36.274906 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 9952
2026-09-28T14:57:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:36 dist-test-slave-pxl6 krb5kdc[10051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:36 dist-test-slave-pxl6 krb5kdc[10051](info): set up 2 sockets
Sep 28 14:57:36 dist-test-slave-pxl6 krb5kdc[10051](info): commencing operation
krb5kdc: starting...
W20260928 14:57:37.307646 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:37 dist-test-slave-pxl6 krb5kdc[10051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:57:37.434077 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:35115
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:35115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:37.517565 10064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:37.517843 10064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:37.517886 10064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:37.519150 10064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:37.519224 10064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:37.519249 10064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:37.519279 10064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:37.519305 10064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:37.520826 10064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:35115
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:35115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:37.521128 10064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:37.521377 10064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:37.526507 10069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:37.526667 10064 server_base.cc:1034] running on GCE node
W20260928 14:57:37.526482 10072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:37.526494 10070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:37.527134 10064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:37.527403 10064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:37.528594 10064 hybrid_clock.cc:648] HybridClock initialized: now 1790607457528581 us; error 31 us; skew 500 ppm
Sep 28 14:57:37 dist-test-slave-pxl6 krb5kdc[10051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607457, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:37.531330 10064 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:57:37.531858 10064 webserver.cc:460] Webserver started at http://127.20.116.190:34563/ using document root <none> and password file <none>
I20260928 14:57:37.532101 10064 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:37.532148 10064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:37.532331 10064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:37.533506 10064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "e6b66506ded84aa096a00878c8746ed1"
format_stamp: "Formatted at 2026-09-28 14:57:37 on dist-test-slave-pxl6"
server_key: "808814d4da070fc5ff1c86164f715bba"
server_key_iv: "b6dcf38177872c6d87da765dfb3dead1"
server_key_version: "encryptionkey@0"
I20260928 14:57:37.533896 10064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "e6b66506ded84aa096a00878c8746ed1"
format_stamp: "Formatted at 2026-09-28 14:57:37 on dist-test-slave-pxl6"
server_key: "808814d4da070fc5ff1c86164f715bba"
server_key_iv: "b6dcf38177872c6d87da765dfb3dead1"
server_key_version: "encryptionkey@0"
I20260928 14:57:37.535429 10064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:37.536270 10079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:37.536468 10064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:57:37.536593 10064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "e6b66506ded84aa096a00878c8746ed1"
format_stamp: "Formatted at 2026-09-28 14:57:37 on dist-test-slave-pxl6"
server_key: "808814d4da070fc5ff1c86164f715bba"
server_key_iv: "b6dcf38177872c6d87da765dfb3dead1"
server_key_version: "encryptionkey@0"
I20260928 14:57:37.536675 10064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:37.551349 10064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:37.553587 10064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:37.553766 10064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:37.558609 10064 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:35115
I20260928 14:57:37.558660 10131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:35115 every 8 connection(s)
I20260928 14:57:37.559067 10064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:37.559373 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 10064
I20260928 14:57:37.559500 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:37.559679 20946 external_mini_cluster.cc:1428] Setting key aaa23efef02d25efd536ac3c655b7190
I20260928 14:57:37.560077 10132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:37.563709 10132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1: Bootstrap starting.
I20260928 14:57:37.564446 10132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:37.564770 10132 log.cc:824] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:37.565749 10132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1: No bootstrap required, opened a new log
Sep 28 14:57:37 dist-test-slave-pxl6 krb5kdc[10051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:57:37.568010 10132 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b66506ded84aa096a00878c8746ed1" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35115 } }
I20260928 14:57:37.568210 10132 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:37.568277 10132 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6b66506ded84aa096a00878c8746ed1, State: Initialized, Role: FOLLOWER
I20260928 14:57:37.568431 10132 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b66506ded84aa096a00878c8746ed1" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35115 } }
I20260928 14:57:37.568536 10132 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:37.568583 10132 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:37.568631 10132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:37.569222 10132 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b66506ded84aa096a00878c8746ed1" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35115 } }
I20260928 14:57:37.569357 10132 leader_election.cc:302] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6b66506ded84aa096a00878c8746ed1; no voters:
I20260928 14:57:37.569581 10132 leader_election.cc:288] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:37.569622 10137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:37.569741 10137 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1 [term 1 LEADER]: Becoming Leader. State: Replica: e6b66506ded84aa096a00878c8746ed1, State: Running, Role: LEADER
I20260928 14:57:37.569878 10132 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:37.569890 10137 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b66506ded84aa096a00878c8746ed1" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35115 } }
I20260928 14:57:37.570364 10139 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6b66506ded84aa096a00878c8746ed1. Latest consensus state: current_term: 1 leader_uuid: "e6b66506ded84aa096a00878c8746ed1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b66506ded84aa096a00878c8746ed1" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35115 } } }
I20260928 14:57:37.570585 10139 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:37.570361 10138 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6b66506ded84aa096a00878c8746ed1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b66506ded84aa096a00878c8746ed1" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 35115 } } }
I20260928 14:57:37.571277 10138 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:37.571805 10135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:37.560244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34173 (local address 127.20.116.190:35115)
0928 14:57:37.560444 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:37.560454 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:37.560489 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:57:37.561013 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:37.561023 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:37.561096 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:37.561268 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:37.562005 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:37.563463 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:37.564331 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:37.565007 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:37.565124 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:37.567726 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:37.567731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:37.567732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:37.567747 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:37.569876 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:37.570388 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:37.570392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:37.570392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:37.570501 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:37.570799 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:37.570800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:37.570800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:37.571086 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:37.571250 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:37.571368 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 14:57:37.571560 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
W20260928 14:57:37.571866 10152 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:57:37.571910 10152 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:57:37.572012 10146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:37.572430 10146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:37.572816 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:57:37.574221 10146 catalog_manager.cc:1347] Generated new cluster ID: 9e79e37fe4cd4c64b4693b09cbac98bf
I20260928 14:57:37.574280 10146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:37.582409 10135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:37.574370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40261 (local address 127.20.116.190:35115)
0928 14:57:37.574407 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:37.574409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:37.574726 (+ 317us) server_negotiation.cc:407] Connection header received
0928 14:57:37.574784 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:37.574784 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:37.574794 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:37.574848 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:37.575527 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:37.576264 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:37.577406 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:37.577731 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:37.577792 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:37.579397 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:37.579399 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:37.579400 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:37.579408 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:37.581064 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:37.581536 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:37.581537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:37.581538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:37.581603 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:37.581846 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:37.581847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:37.581847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:37.581974 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:37.582072 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:37.582246 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 14:57:37.582309 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:57:37.596386 10146 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:37.597000 10146 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:37.612447 10146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6b66506ded84aa096a00878c8746ed1: Generated new TSK 0
I20260928 14:57:37.612718 10146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:37.622192 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 10064
W20260928 14:57:37.627091 10049 connection.cc:535] client connection to 127.20.116.190:35115 recv error: Network error: failed to read from TLS socket (remote: 127.20.116.190:35115): Connection reset by peer (error 104)
2026-09-28T14:57:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1347 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:37 dist-test-slave-pxl6 krb5kdc[10163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:37 dist-test-slave-pxl6 krb5kdc[10163](info): set up 2 sockets
Sep 28 14:57:37 dist-test-slave-pxl6 krb5kdc[10163](info): commencing operation
krb5kdc: starting...
W20260928 14:57:38.679721 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:38 dist-test-slave-pxl6 krb5kdc[10163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:57:38.805518 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:40809
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:38917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:40809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:38.890659 10176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:38.890869 10176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:38.890893 10176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:38.892005 10176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:38.892042 10176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:38.892057 10176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:38.892069 10176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:38.892081 10176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:38.893508 10176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:38917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:40809
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:40809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:38.893713 10176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:38.893887 10176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:38.898793 10184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:38.898825 10182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:38.898833 10181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:38.898947 10176 server_base.cc:1034] running on GCE node
I20260928 14:57:38.899525 10176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:38.899854 10176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:38.901013 10176 hybrid_clock.cc:648] HybridClock initialized: now 1790607458900992 us; error 36 us; skew 500 ppm
Sep 28 14:57:38 dist-test-slave-pxl6 krb5kdc[10163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607458, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:38.908056 10176 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:57:38.908632 10176 webserver.cc:460] Webserver started at http://127.20.116.190:46019/ using document root <none> and password file <none>
I20260928 14:57:38.908885 10176 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:38.908937 10176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:38.909368 10176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:38.910656 10176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "371204d08faa438783b16bec4f3dd274"
format_stamp: "Formatted at 2026-09-28 14:57:38 on dist-test-slave-pxl6"
server_key: "be7cf8c81ef3dceb1c6b04122a945c5b"
server_key_iv: "ec861bbe9544688f82af449106eaef22"
server_key_version: "encryptionkey@0"
I20260928 14:57:38.911038 10176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "371204d08faa438783b16bec4f3dd274"
format_stamp: "Formatted at 2026-09-28 14:57:38 on dist-test-slave-pxl6"
server_key: "be7cf8c81ef3dceb1c6b04122a945c5b"
server_key_iv: "ec861bbe9544688f82af449106eaef22"
server_key_version: "encryptionkey@0"
I20260928 14:57:38.912427 10176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:57:38.913190 10191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:38.913391 10176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:57:38.913482 10176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "371204d08faa438783b16bec4f3dd274"
format_stamp: "Formatted at 2026-09-28 14:57:38 on dist-test-slave-pxl6"
server_key: "be7cf8c81ef3dceb1c6b04122a945c5b"
server_key_iv: "ec861bbe9544688f82af449106eaef22"
server_key_version: "encryptionkey@0"
I20260928 14:57:38.913537 10176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:38.930299 10176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:38.932813 10176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:38.932932 10176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:38.937032 10176 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:40809
I20260928 14:57:38.937089 10243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:40809 every 8 connection(s)
I20260928 14:57:38.937474 10176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:38.938519 10244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:38.941164 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 10176
I20260928 14:57:38.941161 10244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274: Bootstrap starting.
I20260928 14:57:38.941303 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:38.941540 20946 external_mini_cluster.cc:1428] Setting key 9456d2e234d9f6c136412e3800be7671
I20260928 14:57:38.941967 10244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:38.942335 10244 log.cc:824] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:38.943121 10244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274: No bootstrap required, opened a new log
I20260928 14:57:38.944967 10244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371204d08faa438783b16bec4f3dd274" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40809 } }
I20260928 14:57:38.945143 10244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:38.945195 10244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371204d08faa438783b16bec4f3dd274, State: Initialized, Role: FOLLOWER
I20260928 14:57:38.945322 10244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371204d08faa438783b16bec4f3dd274" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40809 } }
I20260928 14:57:38.945395 10244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:38.945417 10244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:38.945484 10244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:38.946027 10244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371204d08faa438783b16bec4f3dd274" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40809 } }
I20260928 14:57:38.946159 10244 leader_election.cc:302] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 371204d08faa438783b16bec4f3dd274; no voters:
I20260928 14:57:38.946339 10244 leader_election.cc:288] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:38.946421 10249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:38.946628 10244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:38.946978 10249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274 [term 1 LEADER]: Becoming Leader. State: Replica: 371204d08faa438783b16bec4f3dd274, State: Running, Role: LEADER
I20260928 14:57:38.947129 10249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371204d08faa438783b16bec4f3dd274" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40809 } }
I20260928 14:57:38.947458 10249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 371204d08faa438783b16bec4f3dd274. Latest consensus state: current_term: 1 leader_uuid: "371204d08faa438783b16bec4f3dd274" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371204d08faa438783b16bec4f3dd274" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40809 } } }
I20260928 14:57:38.947561 10249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:38.947469 10250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "371204d08faa438783b16bec4f3dd274" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371204d08faa438783b16bec4f3dd274" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40809 } } }
I20260928 14:57:38.947727 10250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:38.948141 10247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:38.942167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57941 (local address 127.20.116.190:40809)
0928 14:57:38.942405 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:38.942414 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:38.942464 (+ 50us) server_negotiation.cc:407] Connection header received
0928 14:57:38.942987 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:38.942994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:38.943045 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:38.943204 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:38.943403 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:38.944298 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:38.945152 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:38.945720 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:38.945843 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:38.946740 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:38.946745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:38.946746 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:38.947077 (+ 331us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:38.947121 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:38.947137 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:38.947228 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:38.947530 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 14:57:38.947565 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":95,"threads_started":1}
I20260928 14:57:38.948217 10258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:38.948710 10258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:38.949231 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:57:38.950474 10258 catalog_manager.cc:1347] Generated new cluster ID: f3c17d2b704745358e397420c0ef032e
I20260928 14:57:38.950546 10258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:38.954380 10247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:38.950806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56927 (local address 127.20.116.190:40809)
0928 14:57:38.950840 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:38.950842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:38.950984 (+ 142us) server_negotiation.cc:407] Connection header received
0928 14:57:38.951031 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:38.951032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:38.951043 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:38.951096 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:38.951243 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:38.951980 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:38.952995 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:38.953335 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:38.953411 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:38.954126 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:38.954127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:38.954128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:38.954132 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:38.954149 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:38.954154 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:38.954223 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:38.954292 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 14:57:38.954303 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 14:57:38.971735 10258 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:38.972506 10258 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:38.975739 10258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 371204d08faa438783b16bec4f3dd274: Generated new TSK 0
I20260928 14:57:38.975945 10258 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:38.980592 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 10176
W20260928 14:57:38.985167 10161 connection.cc:535] client connection to 127.20.116.190:40809 recv error: Network error: failed to read from TLS socket (remote: 127.20.116.190:40809): Connection reset by peer (error 104)
2026-09-28T14:57:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:39 dist-test-slave-pxl6 krb5kdc[10275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:39 dist-test-slave-pxl6 krb5kdc[10275](info): set up 2 sockets
Sep 28 14:57:39 dist-test-slave-pxl6 krb5kdc[10275](info): commencing operation
krb5kdc: starting...
W20260928 14:57:40.023756 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:40 dist-test-slave-pxl6 krb5kdc[10275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:57:40.149868 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:40983
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:46053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:40983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:40.233688 10288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:40.233948 10288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:40.233986 10288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:40.235280 10288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:40.235349 10288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:40.235373 10288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:40.235395 10288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:40.235421 10288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:40.236996 10288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:46053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:40983
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:40983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:40.237303 10288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:40.237541 10288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:57:40.242803 10288 server_base.cc:1034] running on GCE node
W20260928 14:57:40.242786 10294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:40.242834 10293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:40.242784 10296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:40.243384 10288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:40.243693 10288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:40.244845 10288 hybrid_clock.cc:648] HybridClock initialized: now 1790607460244830 us; error 41 us; skew 500 ppm
Sep 28 14:57:40 dist-test-slave-pxl6 krb5kdc[10275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607460, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:40.247466 10288 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:57:40.247876 10288 webserver.cc:460] Webserver started at http://127.20.116.190:39565/ using document root <none> and password file <none>
I20260928 14:57:40.248073 10288 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:40.248112 10288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:40.248222 10288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:40.249269 10288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "ae773620ccb5427d82b27c363a76fe77"
format_stamp: "Formatted at 2026-09-28 14:57:40 on dist-test-slave-pxl6"
server_key: "eeda75701abf449824f93b8dfb37b1ff"
server_key_iv: "c18c1b191dcc55f45d2e4ab652f3938c"
server_key_version: "encryptionkey@0"
I20260928 14:57:40.249580 10288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "ae773620ccb5427d82b27c363a76fe77"
format_stamp: "Formatted at 2026-09-28 14:57:40 on dist-test-slave-pxl6"
server_key: "eeda75701abf449824f93b8dfb37b1ff"
server_key_iv: "c18c1b191dcc55f45d2e4ab652f3938c"
server_key_version: "encryptionkey@0"
I20260928 14:57:40.250928 10288 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:57:40.251622 10303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:40.251843 10288 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:57:40.251963 10288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "ae773620ccb5427d82b27c363a76fe77"
format_stamp: "Formatted at 2026-09-28 14:57:40 on dist-test-slave-pxl6"
server_key: "eeda75701abf449824f93b8dfb37b1ff"
server_key_iv: "c18c1b191dcc55f45d2e4ab652f3938c"
server_key_version: "encryptionkey@0"
I20260928 14:57:40.252048 10288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:40.255617 10288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:40.257849 10288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:40.258018 10288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:40.262372 10288 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:40983
I20260928 14:57:40.262428 10355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:40983 every 8 connection(s)
I20260928 14:57:40.262842 10288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:40.263940 10356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:40.264818 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 10288
I20260928 14:57:40.264943 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:40.265146 20946 external_mini_cluster.cc:1428] Setting key c4f05f5a30956eb20ed311a7d11d9bd5
I20260928 14:57:40.266701 10356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77: Bootstrap starting.
I20260928 14:57:40.267467 10356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:40.267810 10356 log.cc:824] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:40.267886 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:40.265711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41413 (local address 127.20.116.190:40983)
0928 14:57:40.265907 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:40.265914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:40.265940 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:57:40.266399 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:40.266405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:40.266431 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:40.266621 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:40.267399 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:40.267402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:40.267402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:40.267406 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:40.267445 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:40.267455 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:40.267544 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:40.267604 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 14:57:40.267666 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":102,"threads_started":1}
I20260928 14:57:40.268656 10356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77: No bootstrap required, opened a new log
I20260928 14:57:40.268931 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:57:40.271685 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:40.270550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53267 (local address 127.20.116.190:40983)
0928 14:57:40.270591 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:40.270593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:40.270743 (+ 150us) server_negotiation.cc:407] Connection header received
0928 14:57:40.270765 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:40.270766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:40.270772 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:40.270838 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:40.271479 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:40.271480 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:40.271481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:40.271485 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:40.271499 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:40.271503 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:40.271566 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:40.271625 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 14:57:40.271637 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:57:40.271588 10356 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae773620ccb5427d82b27c363a76fe77" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40983 } }
I20260928 14:57:40.271766 10356 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:40.271804 10356 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae773620ccb5427d82b27c363a76fe77, State: Initialized, Role: FOLLOWER
I20260928 14:57:40.271957 10356 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae773620ccb5427d82b27c363a76fe77" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40983 } }
I20260928 14:57:40.272076 10356 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:40.272128 10356 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:40.272176 10356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:40.272773 10356 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae773620ccb5427d82b27c363a76fe77" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40983 } }
I20260928 14:57:40.272876 10356 leader_election.cc:302] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae773620ccb5427d82b27c363a76fe77; no voters:
I20260928 14:57:40.273002 10356 leader_election.cc:288] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:40.273069 10366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:40.273240 10356 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:40.273247 10366 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77 [term 1 LEADER]: Becoming Leader. State: Replica: ae773620ccb5427d82b27c363a76fe77, State: Running, Role: LEADER
I20260928 14:57:40.273370 10366 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae773620ccb5427d82b27c363a76fe77" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40983 } }
I20260928 14:57:40.273890 10368 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae773620ccb5427d82b27c363a76fe77. Latest consensus state: current_term: 1 leader_uuid: "ae773620ccb5427d82b27c363a76fe77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae773620ccb5427d82b27c363a76fe77" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40983 } } }
I20260928 14:57:40.273905 10367 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae773620ccb5427d82b27c363a76fe77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae773620ccb5427d82b27c363a76fe77" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 40983 } } }
I20260928 14:57:40.274004 10368 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:40.274024 10367 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:40.274209 10375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:40.274749 10375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:40.276384 10375 catalog_manager.cc:1347] Generated new cluster ID: a390dfd3231046e5843b32f3805e93ba
I20260928 14:57:40.276428 10375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:40.284823 10375 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:40.285012 10375 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:40.288697 10375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae773620ccb5427d82b27c363a76fe77: Generated new TSK 0
I20260928 14:57:40.288925 10375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:40.291659 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 10288
2026-09-28T14:57:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:40 dist-test-slave-pxl6 krb5kdc[10387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:40 dist-test-slave-pxl6 krb5kdc[10387](info): set up 2 sockets
Sep 28 14:57:40 dist-test-slave-pxl6 krb5kdc[10387](info): commencing operation
krb5kdc: starting...
W20260928 14:57:41.339661 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:41 dist-test-slave-pxl6 krb5kdc[10387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:57:41.465703 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:42257
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:40669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:42257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:41.548659 10400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:41.548879 10400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:41.548904 10400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:41.550019 10400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:41.550055 10400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:41.550071 10400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:41.550083 10400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:41.550096 10400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:41.551707 10400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:40669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:42257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:42257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:41.551996 10400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:41.552253 10400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:41.557329 10408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:41.557431 10406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:41.557334 10405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:41.557657 10400 server_base.cc:1034] running on GCE node
I20260928 14:57:41.557945 10400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:41.558255 10400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:41.559407 10400 hybrid_clock.cc:648] HybridClock initialized: now 1790607461559385 us; error 46 us; skew 500 ppm
Sep 28 14:57:41 dist-test-slave-pxl6 krb5kdc[10387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607461, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:41.562075 10400 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:57:41.562618 10400 webserver.cc:460] Webserver started at http://127.20.116.190:38151/ using document root <none> and password file <none>
I20260928 14:57:41.562866 10400 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:41.562937 10400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:41.563099 10400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:41.564230 10400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "bebc7d90ec064007a2275a6870e19b45"
format_stamp: "Formatted at 2026-09-28 14:57:41 on dist-test-slave-pxl6"
server_key: "dea1078a93c31506b4bde579736797e8"
server_key_iv: "83ae07f9be729ff429e083e042c89388"
server_key_version: "encryptionkey@0"
I20260928 14:57:41.564615 10400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "bebc7d90ec064007a2275a6870e19b45"
format_stamp: "Formatted at 2026-09-28 14:57:41 on dist-test-slave-pxl6"
server_key: "dea1078a93c31506b4bde579736797e8"
server_key_iv: "83ae07f9be729ff429e083e042c89388"
server_key_version: "encryptionkey@0"
I20260928 14:57:41.566027 10400 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:57:41.566891 10415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:41.567075 10400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:57:41.567200 10400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "bebc7d90ec064007a2275a6870e19b45"
format_stamp: "Formatted at 2026-09-28 14:57:41 on dist-test-slave-pxl6"
server_key: "dea1078a93c31506b4bde579736797e8"
server_key_iv: "83ae07f9be729ff429e083e042c89388"
server_key_version: "encryptionkey@0"
I20260928 14:57:41.567281 10400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:41.594483 10400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:41.596621 10400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:41.596794 10400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:41.601065 10400 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:42257
I20260928 14:57:41.601096 10467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:42257 every 8 connection(s)
I20260928 14:57:41.601459 10400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:41.602293 10468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:41.604645 10468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45: Bootstrap starting.
I20260928 14:57:41.605299 10468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:41.605600 10468 log.cc:824] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:41.606408 10468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45: No bootstrap required, opened a new log
I20260928 14:57:41.608083 10468 raft_consensus.cc:348] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bebc7d90ec064007a2275a6870e19b45" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42257 } }
I20260928 14:57:41.608253 10468 raft_consensus.cc:374] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:41.608307 10468 raft_consensus.cc:729] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebc7d90ec064007a2275a6870e19b45, State: Initialized, Role: FOLLOWER
I20260928 14:57:41.608482 10468 consensus_queue.cc:260] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bebc7d90ec064007a2275a6870e19b45" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42257 } }
I20260928 14:57:41.608570 10468 raft_consensus.cc:388] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:41.608615 10468 raft_consensus.cc:482] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:41.608667 10468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:41.609654 10468 raft_consensus.cc:504] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bebc7d90ec064007a2275a6870e19b45" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42257 } }
I20260928 14:57:41.609778 10468 leader_election.cc:302] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bebc7d90ec064007a2275a6870e19b45; no voters:
I20260928 14:57:41.609965 10468 leader_election.cc:288] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:41.610013 10471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:41.610240 10468 sys_catalog.cc:564] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:41.610281 10471 raft_consensus.cc:686] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45 [term 1 LEADER]: Becoming Leader. State: Replica: bebc7d90ec064007a2275a6870e19b45, State: Running, Role: LEADER
I20260928 14:57:41.610416 10471 consensus_queue.cc:237] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bebc7d90ec064007a2275a6870e19b45" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42257 } }
I20260928 14:57:41.610878 10473 sys_catalog.cc:455] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bebc7d90ec064007a2275a6870e19b45. Latest consensus state: current_term: 1 leader_uuid: "bebc7d90ec064007a2275a6870e19b45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bebc7d90ec064007a2275a6870e19b45" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42257 } } }
I20260928 14:57:41.610970 10473 sys_catalog.cc:458] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:41.611146 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 10400
I20260928 14:57:41.611253 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:41.611322 10472 sys_catalog.cc:455] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bebc7d90ec064007a2275a6870e19b45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bebc7d90ec064007a2275a6870e19b45" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42257 } } }
I20260928 14:57:41.611392 10472 sys_catalog.cc:458] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:41.611477 20946 external_mini_cluster.cc:1428] Setting key f48b2da0b9e93f2c9e97cf53594dbdc2
I20260928 14:57:41.611687 10480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:41.612294 10480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:41.613926 10480 catalog_manager.cc:1347] Generated new cluster ID: 29c95603a6f545eeb6ea75e3a589a005
I20260928 14:57:41.613982 10480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:57:41 dist-test-slave-pxl6 krb5kdc[10387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:57:41.623015 10480 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:41.623309 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:41.612001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40699 (local address 127.20.116.190:42257)
0928 14:57:41.612385 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:41.612397 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:41.612601 (+ 204us) server_negotiation.cc:407] Connection header received
0928 14:57:41.612870 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:41.612876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:41.612955 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:41.613096 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:41.614685 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:41.615637 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:41.616437 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:41.616903 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:41.617042 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:41.619489 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:41.619497 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:41.619497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:41.619510 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:41.621270 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:41.621672 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:41.621675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:41.621676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:41.621835 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:41.622059 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:41.622060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:41.622061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:41.622402 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:41.622600 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:41.622897 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 14:57:41.623090 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":66,"threads_started":1}
I20260928 14:57:41.623756 10480 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:41.624253 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:57:41.633071 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:41.625778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55267 (local address 127.20.116.190:42257)
0928 14:57:41.625828 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:41.625830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:41.626742 (+ 912us) server_negotiation.cc:407] Connection header received
0928 14:57:41.626862 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:41.626863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:41.626875 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:41.626931 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:41.627476 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:41.627991 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:41.628767 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:41.628985 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:41.629063 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:41.630364 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:41.630366 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:41.630366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:41.630374 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:41.631863 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:41.632226 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:41.632227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:41.632228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:41.632298 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:41.632501 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:41.632502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:41.632503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:41.632635 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:41.632740 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:41.632873 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 14:57:41.632958 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 14:57:41.635342 10480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bebc7d90ec064007a2275a6870e19b45: Generated new TSK 0
I20260928 14:57:41.635545 10480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:41.638767 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 10400
2026-09-28T14:57:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1347 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:41 dist-test-slave-pxl6 krb5kdc[10499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:41 dist-test-slave-pxl6 krb5kdc[10499](info): set up 2 sockets
Sep 28 14:57:41 dist-test-slave-pxl6 krb5kdc[10499](info): commencing operation
krb5kdc: starting...
W20260928 14:57:42.675709 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:42 dist-test-slave-pxl6 krb5kdc[10499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:57:42.799925 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:42253
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:32909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:42253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:42.885380 10512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:42.885601 10512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:42.885625 10512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:42.886876 10512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:42.886919 10512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:42.886934 10512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:42.886947 10512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:42.886960 10512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:42.888489 10512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:32909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:42253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:42253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:42.888767 10512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:42.889015 10512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:57:42.894119 10512 server_base.cc:1034] running on GCE node
W20260928 14:57:42.894032 10518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:42.894161 10517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:42.894032 10520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:42.894502 10512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:42.894891 10512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:42.896035 10512 hybrid_clock.cc:648] HybridClock initialized: now 1790607462896023 us; error 31 us; skew 500 ppm
Sep 28 14:57:42 dist-test-slave-pxl6 krb5kdc[10499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607462, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:42.898758 10512 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:57:42.899216 10512 webserver.cc:460] Webserver started at http://127.20.116.190:35771/ using document root <none> and password file <none>
I20260928 14:57:42.899473 10512 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:42.899519 10512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:42.899657 10512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:42.900719 10512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "4eac13604cb04b01bd2e7cb3af12983a"
format_stamp: "Formatted at 2026-09-28 14:57:42 on dist-test-slave-pxl6"
server_key: "7fe00ca138e0a6bbd190aa0340e0e55d"
server_key_iv: "df51408defee6f927d527ad0551c3d8b"
server_key_version: "encryptionkey@0"
I20260928 14:57:42.901082 10512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "4eac13604cb04b01bd2e7cb3af12983a"
format_stamp: "Formatted at 2026-09-28 14:57:42 on dist-test-slave-pxl6"
server_key: "7fe00ca138e0a6bbd190aa0340e0e55d"
server_key_iv: "df51408defee6f927d527ad0551c3d8b"
server_key_version: "encryptionkey@0"
I20260928 14:57:42.902525 10512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:42.903352 10527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:42.903512 10512 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:57:42.903621 10512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "4eac13604cb04b01bd2e7cb3af12983a"
format_stamp: "Formatted at 2026-09-28 14:57:42 on dist-test-slave-pxl6"
server_key: "7fe00ca138e0a6bbd190aa0340e0e55d"
server_key_iv: "df51408defee6f927d527ad0551c3d8b"
server_key_version: "encryptionkey@0"
I20260928 14:57:42.903698 10512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:42.913314 10512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:42.915722 10512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:42.915898 10512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:42.920109 10512 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:42253
I20260928 14:57:42.920164 10579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:42253 every 8 connection(s)
I20260928 14:57:42.920583 10512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:42.921725 10580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:42.924499 10580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a: Bootstrap starting.
I20260928 14:57:42.925182 10580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:42.925307 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 10512
I20260928 14:57:42.925453 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:42.925612 10580 log.cc:824] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:42.925732 20946 external_mini_cluster.cc:1428] Setting key 55ca268b12ca8c91fbba80296acacf77
I20260928 14:57:42.926565 10580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a: No bootstrap required, opened a new log
I20260928 14:57:42.929188 10580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eac13604cb04b01bd2e7cb3af12983a" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42253 } }
I20260928 14:57:42.929371 10580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:42.929427 10580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eac13604cb04b01bd2e7cb3af12983a, State: Initialized, Role: FOLLOWER
I20260928 14:57:42.929567 10580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eac13604cb04b01bd2e7cb3af12983a" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42253 } }
I20260928 14:57:42.929661 10580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:42.929709 10580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:42.929764 10580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:42.930397 10580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eac13604cb04b01bd2e7cb3af12983a" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42253 } }
I20260928 14:57:42.930498 10580 leader_election.cc:302] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4eac13604cb04b01bd2e7cb3af12983a; no voters:
I20260928 14:57:42.930795 10580 leader_election.cc:288] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:42.930836 10585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:42.931008 10585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a [term 1 LEADER]: Becoming Leader. State: Replica: 4eac13604cb04b01bd2e7cb3af12983a, State: Running, Role: LEADER
I20260928 14:57:42.931181 10580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:42.931165 10585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eac13604cb04b01bd2e7cb3af12983a" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42253 } }
I20260928 14:57:42.931712 10586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4eac13604cb04b01bd2e7cb3af12983a. Latest consensus state: current_term: 1 leader_uuid: "4eac13604cb04b01bd2e7cb3af12983a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eac13604cb04b01bd2e7cb3af12983a" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42253 } } }
I20260928 14:57:42.931844 10586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:42.932082 10593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:42.932294 10587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4eac13604cb04b01bd2e7cb3af12983a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eac13604cb04b01bd2e7cb3af12983a" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 42253 } } }
I20260928 14:57:42.932381 10587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:42.932582 10584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:42.926336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51513 (local address 127.20.116.190:42253)
0928 14:57:42.926572 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:42.926583 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:42.926628 (+ 45us) server_negotiation.cc:407] Connection header received
0928 14:57:42.927140 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:42.927147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:42.927200 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:42.927361 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:42.927559 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:42.928457 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:42.929386 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:42.929967 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:42.930101 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:42.930914 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:42.930919 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:42.930919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:42.931170 (+ 251us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:42.931218 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:42.931232 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:42.931470 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:42.931507 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 14:57:42.931579 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":89,"threads_started":1}
I20260928 14:57:42.932590 10593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:42.933801 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:57:42.934453 10593 catalog_manager.cc:1347] Generated new cluster ID: 16d176d883ac4b6286c3c0a371201120
I20260928 14:57:42.934495 10593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:42.939258 10584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:42.935473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48939 (local address 127.20.116.190:42253)
0928 14:57:42.935518 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:42.935520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:42.935648 (+ 128us) server_negotiation.cc:407] Connection header received
0928 14:57:42.935680 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:42.935681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:42.935692 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:42.935773 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:42.935888 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:42.936715 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:42.937763 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:42.938122 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:42.938204 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:42.938990 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:42.938992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:42.938992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:42.938999 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:42.939016 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:42.939021 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:42.939093 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:42.939148 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 14:57:42.939175 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:57:42.944211 10593 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:42.944883 10593 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:42.950986 10593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4eac13604cb04b01bd2e7cb3af12983a: Generated new TSK 0
I20260928 14:57:42.951215 10593 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:42.958979 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 10512
2026-09-28T14:57:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:43 dist-test-slave-pxl6 krb5kdc[10611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:43 dist-test-slave-pxl6 krb5kdc[10611](info): set up 2 sockets
Sep 28 14:57:43 dist-test-slave-pxl6 krb5kdc[10611](info): commencing operation
krb5kdc: starting...
W20260928 14:57:44.023674 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:44 dist-test-slave-pxl6 krb5kdc[10611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:57:44.150450 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:36813
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:40799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:36813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:44.234087 10624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:44.234321 10624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:44.234346 10624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:44.235697 10624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:44.235744 10624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:44.235759 10624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:44.235772 10624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:44.235785 10624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:44.237360 10624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:40799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:36813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:36813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:44.237635 10624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:44.237884 10624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:44.243099 10632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:44.243130 10630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:44.243284 10624 server_base.cc:1034] running on GCE node
W20260928 14:57:44.243125 10629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:44.243593 10624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:44.243940 10624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:44.245107 10624 hybrid_clock.cc:648] HybridClock initialized: now 1790607464245098 us; error 44 us; skew 500 ppm
Sep 28 14:57:44 dist-test-slave-pxl6 krb5kdc[10611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607464, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:44.254276 10624 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:57:44.254942 10624 webserver.cc:460] Webserver started at http://127.20.116.190:44419/ using document root <none> and password file <none>
I20260928 14:57:44.255198 10624 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:44.255270 10624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:44.255424 10624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:44.256641 10624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "dd708ef3ce4f46a88c97f62d682314ed"
format_stamp: "Formatted at 2026-09-28 14:57:44 on dist-test-slave-pxl6"
server_key: "fbad4d8a82ca8ccca8140bd471c3bd89"
server_key_iv: "28463cb471252b49f944eaa843e4397c"
server_key_version: "encryptionkey@0"
I20260928 14:57:44.257037 10624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "dd708ef3ce4f46a88c97f62d682314ed"
format_stamp: "Formatted at 2026-09-28 14:57:44 on dist-test-slave-pxl6"
server_key: "fbad4d8a82ca8ccca8140bd471c3bd89"
server_key_iv: "28463cb471252b49f944eaa843e4397c"
server_key_version: "encryptionkey@0"
I20260928 14:57:44.258509 10624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:44.259473 10639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:44.259711 10624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:57:44.259825 10624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "dd708ef3ce4f46a88c97f62d682314ed"
format_stamp: "Formatted at 2026-09-28 14:57:44 on dist-test-slave-pxl6"
server_key: "fbad4d8a82ca8ccca8140bd471c3bd89"
server_key_iv: "28463cb471252b49f944eaa843e4397c"
server_key_version: "encryptionkey@0"
I20260928 14:57:44.259915 10624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:44.263479 10624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:44.265635 10624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:44.265810 10624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:44.270080 10624 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:36813
I20260928 14:57:44.270116 10691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:36813 every 8 connection(s)
I20260928 14:57:44.270475 10624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:44.271535 10692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:44.274495 10692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed: Bootstrap starting.
I20260928 14:57:44.275223 10692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:44.275521 10692 log.cc:824] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:44.275866 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 10624
I20260928 14:57:44.276012 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:44.276208 20946 external_mini_cluster.cc:1428] Setting key d18767a0a8e0a6e6823e21fe5be997a3
I20260928 14:57:44.276392 10692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed: No bootstrap required, opened a new log
I20260928 14:57:44.278951 10692 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd708ef3ce4f46a88c97f62d682314ed" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36813 } }
I20260928 14:57:44.279135 10692 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:44.279165 10692 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd708ef3ce4f46a88c97f62d682314ed, State: Initialized, Role: FOLLOWER
I20260928 14:57:44.279273 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:44.276803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46051 (local address 127.20.116.190:36813)
0928 14:57:44.276991 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:44.276999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:44.277024 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:57:44.277805 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:44.277814 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:44.277844 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:44.277968 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:44.278682 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:44.278685 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:44.278685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:44.278689 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:44.278727 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:44.278738 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:44.278801 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:44.278893 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 14:57:44.278928 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260928 14:57:44.279291 10692 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd708ef3ce4f46a88c97f62d682314ed" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36813 } }
I20260928 14:57:44.279429 10692 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:44.279469 10692 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:44.279520 10692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:44.280325 10692 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd708ef3ce4f46a88c97f62d682314ed" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36813 } }
I20260928 14:57:44.280401 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:57:44.280439 10692 leader_election.cc:302] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd708ef3ce4f46a88c97f62d682314ed; no voters:
I20260928 14:57:44.280630 10692 leader_election.cc:288] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:44.280651 10697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:44.280773 10697 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed [term 1 LEADER]: Becoming Leader. State: Replica: dd708ef3ce4f46a88c97f62d682314ed, State: Running, Role: LEADER
I20260928 14:57:44.280911 10692 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:44.280920 10697 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd708ef3ce4f46a88c97f62d682314ed" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36813 } }
I20260928 14:57:44.281396 10699 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd708ef3ce4f46a88c97f62d682314ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd708ef3ce4f46a88c97f62d682314ed" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36813 } } }
I20260928 14:57:44.281500 10699 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:44.281716 10698 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd708ef3ce4f46a88c97f62d682314ed. Latest consensus state: current_term: 1 leader_uuid: "dd708ef3ce4f46a88c97f62d682314ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd708ef3ce4f46a88c97f62d682314ed" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36813 } } }
I20260928 14:57:44.282048 10698 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:44.282244 10708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:44.282786 10708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:44.283362 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:44.281991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59063 (local address 127.20.116.190:36813)
0928 14:57:44.282028 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:44.282030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:44.282254 (+ 224us) server_negotiation.cc:407] Connection header received
0928 14:57:44.282353 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:44.282354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:44.282359 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:44.282389 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:44.283117 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:44.283118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:44.283119 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:57:44.283123 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:44.283137 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:57:44.283142 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:44.283208 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:44.283267 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 14:57:44.283281 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:57:44.284340 10708 catalog_manager.cc:1347] Generated new cluster ID: d830152d8d6643f086c66d353fedb81c
I20260928 14:57:44.284399 10708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:44.296705 10708 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:44.296909 10708 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:44.302117 10708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd708ef3ce4f46a88c97f62d682314ed: Generated new TSK 0
I20260928 14:57:44.302356 10708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:44.305429 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 10624
2026-09-28T14:57:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1345 ms)
[----------] 18 tests from AuthTokenIssuingTest (24152 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:44 dist-test-slave-pxl6 krb5kdc[10723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:44 dist-test-slave-pxl6 krb5kdc[10723](info): set up 2 sockets
Sep 28 14:57:44 dist-test-slave-pxl6 krb5kdc[10723](info): commencing operation
krb5kdc: starting...
W20260928 14:57:45.343730 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:45 dist-test-slave-pxl6 krb5kdc[10723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:57:45.469055 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:39513
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:39513,127.20.116.189:42873,127.20.116.188:43173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:45.550516 10736 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:57:45.550770 10736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:45.550787 10736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:45.550807 10736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:45.552103 10736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:45.552146 10736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:45.552160 10736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:45.552172 10736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:45.552186 10736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:45.553660 10736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:39513,127.20.116.189:42873,127.20.116.188:43173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:39513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:45.553913 10736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:45.554131 10736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:45.559502 10741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:45.559473 10742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:45.559564 10736 server_base.cc:1034] running on GCE node
W20260928 14:57:45.559461 10744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:45.559911 10736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:45.560236 10736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:45.561391 10736 hybrid_clock.cc:648] HybridClock initialized: now 1790607465561376 us; error 29 us; skew 500 ppm
Sep 28 14:57:45 dist-test-slave-pxl6 krb5kdc[10723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607465, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:45.564193 10736 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:57:45.564638 10736 webserver.cc:460] Webserver started at http://127.20.116.190:34967/ using document root <none> and password file <none>
I20260928 14:57:45.564896 10736 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:45.564967 10736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:45.565115 10736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:45.566200 10736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "ed2839effc894d069a1dc67cb741b89d"
format_stamp: "Formatted at 2026-09-28 14:57:45 on dist-test-slave-pxl6"
server_key: "1b0a0be0102537515b0ca729c289d96b"
server_key_iv: "73cafd0f549bfd2faf43a0105bc3fb98"
server_key_version: "encryptionkey@0"
I20260928 14:57:45.566628 10736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "ed2839effc894d069a1dc67cb741b89d"
format_stamp: "Formatted at 2026-09-28 14:57:45 on dist-test-slave-pxl6"
server_key: "1b0a0be0102537515b0ca729c289d96b"
server_key_iv: "73cafd0f549bfd2faf43a0105bc3fb98"
server_key_version: "encryptionkey@0"
I20260928 14:57:45.568022 10736 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:57:45.568866 10751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:45.569070 10736 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:57:45.569171 10736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "ed2839effc894d069a1dc67cb741b89d"
format_stamp: "Formatted at 2026-09-28 14:57:45 on dist-test-slave-pxl6"
server_key: "1b0a0be0102537515b0ca729c289d96b"
server_key_iv: "73cafd0f549bfd2faf43a0105bc3fb98"
server_key_version: "encryptionkey@0"
I20260928 14:57:45.569257 10736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:45.587471 10736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:45.589907 10736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:45.590075 10736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:45.594336 10736 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:39513
I20260928 14:57:45.594378 10803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:39513 every 8 connection(s)
I20260928 14:57:45.594794 10736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:45.595752 10804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:45.597317 10804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39513 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:57:45 dist-test-slave-pxl6 krb5kdc[10723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607465, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:57:45.604055 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 10736
I20260928 14:57:45.604164 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:45.604377 20946 external_mini_cluster.cc:1428] Setting key 312021ca3a0f1d7b71268d03e8a3f341
I20260928 14:57:45.607331 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.597964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:39513 (local address 127.0.0.1:38222)
0928 14:57:45.598165 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.598209 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:45.598676 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:45.598991 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:45.598993 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:45.599250 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:45.599410 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:45.599415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.600425 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.600426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:45.601086 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:45.601087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.601563 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.601563 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:45.601623 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.602195 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:45.602204 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:45.603673 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:45.605763 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:45.605770 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:45.605771 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:45.605962 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:45.606272 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:45.606272 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:45.606273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:45.606354 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:45.606851 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:45.606858 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:45.606922 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 14:57:45.607013 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 14:57:45.607155 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
I20260928 14:57:45.607429 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.598047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38222 (local address 127.20.116.190:39513)
0928 14:57:45.598251 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:45.598256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:45.598286 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:57:45.598855 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:45.598857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:45.598875 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:45.598964 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:45.599507 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.600342 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.601164 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.601484 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.601568 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.603769 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:45.603778 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:45.603778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:45.603784 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:45.605670 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.606077 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.606080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.606081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.606197 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.606420 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.606421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.606421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.606760 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:45.606849 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:45.607084 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 14:57:45.607244 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":166,"thread_start_us":132,"threads_started":1}
I20260928 14:57:45.608594 10804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:57:45.608965 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.608847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:42873 (local address 127.0.0.1:53300)
0928 14:57:45.608896 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.608947 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.189:42873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 14:57:45.609139 10753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.116.189:42873: connect: Connection refused (error 111)
I20260928 14:57:45.609614 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.609508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:42873 (local address 127.0.0.1:53316)
0928 14:57:45.609564 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.609598 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.189:42873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 14:57:45.609743 10804 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.189:42873: Network error: Client connection negotiation failed: client connection to 127.20.116.189:42873: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.20.116.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.189.keytab.
Entry for principal kudu/127.20.116.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.189.keytab.
I20260928 14:57:45.630125 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.189:42873
--webserver_interface=127.20.116.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.189.keytab
--principal=kudu/127.20.116.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:39513,127.20.116.189:42873,127.20.116.188:43173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:57:45.658993 10804 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } attempt: 1
I20260928 14:57:45.659567 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.659379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:42873 (local address 127.0.0.1:53318)
0928 14:57:45.659451 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.659499 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.189:42873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 14:57:45.659982 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.659886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:42873 (local address 127.0.0.1:53322)
0928 14:57:45.659932 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.659964 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.189:42873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 14:57:45.660073 10804 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.189:42873: Network error: Client connection negotiation failed: client connection to 127.20.116.189:42873: connect: Connection refused (error 111)
W20260928 14:57:45.716784 10811 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:57:45.717027 10811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:45.717067 10811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:45.717108 10811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:45.718286 10811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:45.718355 10811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:45.718382 10811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:45.718413 10811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:45.718439 10811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:45.720022 10811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:39513,127.20.116.189:42873,127.20.116.188:43173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.189.keytab
--principal=kudu/127.20.116.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.189:42873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.116.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:45.720314 10811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:45.720551 10811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:45.725589 10817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:45.725757 10816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:45.725692 10811 server_base.cc:1034] running on GCE node
W20260928 14:57:45.725585 10819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:45.726115 10811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:45.726428 10811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:45.727610 10811 hybrid_clock.cc:648] HybridClock initialized: now 1790607465727592 us; error 51 us; skew 500 ppm
I20260928 14:57:45.728232 10804 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } attempt: 2
I20260928 14:57:45.728639 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.728501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:42873 (local address 127.0.0.1:53334)
0928 14:57:45.728568 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.728610 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.189:42873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 14:57:45.729043 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.728957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:42873 (local address 127.0.0.1:53338)
0928 14:57:45.729001 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.729026 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.189:42873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 14:57:45.729154 10804 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.189:42873: Network error: Client connection negotiation failed: client connection to 127.20.116.189:42873: connect: Connection refused (error 111)
Sep 28 14:57:45 dist-test-slave-pxl6 krb5kdc[10723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607465, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:45.730660 10811 init.cc:377] Logged in from keytab as kudu/127.20.116.189@KRBTEST.COM (short username kudu)
I20260928 14:57:45.731308 10811 webserver.cc:460] Webserver started at http://127.20.116.189:37913/ using document root <none> and password file <none>
I20260928 14:57:45.731556 10811 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:45.731626 10811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:45.731765 10811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:45.732864 10811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-1/data/instance:
uuid: "4254c5510cb340c09fe6e8c08be95bcf"
format_stamp: "Formatted at 2026-09-28 14:57:45 on dist-test-slave-pxl6"
server_key: "54f8d25ad6cc014f78ebedc20ca2284f"
server_key_iv: "51e9edc36c6eb68b98afeec1e815634e"
server_key_version: "encryptionkey@0"
I20260928 14:57:45.733244 10811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-1/wal/instance:
uuid: "4254c5510cb340c09fe6e8c08be95bcf"
format_stamp: "Formatted at 2026-09-28 14:57:45 on dist-test-slave-pxl6"
server_key: "54f8d25ad6cc014f78ebedc20ca2284f"
server_key_iv: "51e9edc36c6eb68b98afeec1e815634e"
server_key_version: "encryptionkey@0"
I20260928 14:57:45.734635 10811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:45.735387 10826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:45.735566 10811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:57:45.735666 10811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-1/wal
uuid: "4254c5510cb340c09fe6e8c08be95bcf"
format_stamp: "Formatted at 2026-09-28 14:57:45 on dist-test-slave-pxl6"
server_key: "54f8d25ad6cc014f78ebedc20ca2284f"
server_key_iv: "51e9edc36c6eb68b98afeec1e815634e"
server_key_version: "encryptionkey@0"
I20260928 14:57:45.735805 10811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:45.746934 10811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:45.749265 10811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:45.749378 10811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:45.753644 10811 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.189:42873
I20260928 14:57:45.753679 10878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.189:42873 every 8 connection(s)
I20260928 14:57:45.754053 10811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-1/data/info.pb
I20260928 14:57:45.755054 10879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:45.755326 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 10811
I20260928 14:57:45.755405 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-1/wal/instance
I20260928 14:57:45.755653 20946 external_mini_cluster.cc:1428] Setting key 7ed2f870fce62b6552c1c7e826880265
I20260928 14:57:45.756861 10879 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39513 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:57:45 dist-test-slave-pxl6 krb5kdc[10723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607465, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.189@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.116.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.188@KRBTEST.COM" created.
I20260928 14:57:45.766047 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.757646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.20.116.190:39513)
0928 14:57:45.757706 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:45.757709 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:45.757931 (+ 222us) server_negotiation.cc:407] Connection header received
0928 14:57:45.758467 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:45.758469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:45.758486 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:45.758584 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:45.759257 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.759758 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.760782 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.761007 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.761083 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.763335 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:45.763336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:45.763336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:45.763345 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:45.764622 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.765038 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.765039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.765039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.765101 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.765351 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.765352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.765353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.765480 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:45.765560 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:45.765864 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 14:57:45.765952 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 14:57:45.766245 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.757636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:39513 (local address 127.0.0.1:38230)
0928 14:57:45.757804 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.757842 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:45.758348 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:45.758655 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:45.758659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:45.758940 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:45.759147 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:45.759152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.759857 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.759858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:45.760666 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:45.760669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.761105 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.761107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:45.761235 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.761834 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:45.761843 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:45.763234 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:45.764718 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:45.764736 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:45.764738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:45.764960 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:45.765169 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:45.765170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:45.765170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:45.765280 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:45.765550 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:45.765558 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:45.765699 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 14:57:45.765833 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 14:57:45.765994 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260928 14:57:45.767113 10879 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:57:45 dist-test-slave-pxl6 krb5kdc[10723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607465, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.189@KRBTEST.COM for kudu/127.20.116.189@KRBTEST.COM
I20260928 14:57:45.774930 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.767413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:42873 (local address 127.0.0.1:53352)
0928 14:57:45.767604 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.767615 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:45.767663 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:45.767822 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:45.767823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:45.767961 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:45.768006 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:45.768008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.768885 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.768886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:45.769446 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:45.769448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.769933 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.769934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:45.770000 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.770496 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:45.770501 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:45.771735 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:45.773897 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:45.773898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:45.773899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:45.774057 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:45.774266 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:45.774267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:45.774267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:45.774316 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:45.774614 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:45.774615 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:45.774680 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 14:57:45.774775 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 14:57:45.774859 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260928 14:57:45.775002 10884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.767460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53352 (local address 127.20.116.189:42873)
0928 14:57:45.767617 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:45.767623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:45.767662 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:57:45.767728 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:45.767731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:45.767744 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:45.767816 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:45.768093 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.768779 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.769509 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.769843 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.769935 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.771854 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:45.771864 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:45.771864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:45.771872 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:45.773798 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.774133 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.774137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.774137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.774200 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.774356 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.774356 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.774356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.774497 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:45.774689 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:45.774810 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 14:57:45.774908 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260928 14:57:45.775681 10879 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:57:45.776067 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.775959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:43173 (local address 127.0.0.1:34694)
0928 14:57:45.776007 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.776049 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 14:57:45.776224 10827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.116.188:43173: connect: Connection refused (error 111)
I20260928 14:57:45.776640 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.776552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:43173 (local address 127.0.0.1:34704)
0928 14:57:45.776601 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.776625 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 14:57:45.776774 10879 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.188:43173: Network error: Client connection negotiation failed: client connection to 127.20.116.188:43173: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.188.keytab.
Entry for principal kudu/127.20.116.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.188.keytab.
I20260928 14:57:45.781747 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.188:43173
--webserver_interface=127.20.116.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.188.keytab
--principal=kudu/127.20.116.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:39513,127.20.116.189:42873,127.20.116.188:43173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:57:45.820331 10804 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } attempt: 3
Sep 28 14:57:45 dist-test-slave-pxl6 krb5kdc[10723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607465, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for kudu/127.20.116.189@KRBTEST.COM
I20260928 14:57:45.826004 10879 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } attempt: 1
I20260928 14:57:45.826416 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.826268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:43173 (local address 127.0.0.1:34720)
0928 14:57:45.826332 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.826374 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 14:57:45.827427 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.827295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:43173 (local address 127.0.0.1:34730)
0928 14:57:45.827347 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.827378 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 14:57:45.827569 10879 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.188:43173: Network error: Client connection negotiation failed: client connection to 127.20.116.188:43173: connect: Connection refused (error 111)
I20260928 14:57:45.828261 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.820762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:42873 (local address 127.0.0.1:53368)
0928 14:57:45.820846 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.820863 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:45.820939 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:45.821094 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:45.821095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:45.821301 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:45.821386 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:45.821389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.822092 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.822093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:45.822950 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:45.822954 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.823412 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.823413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:45.823493 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.823966 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:45.823971 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:45.825323 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:45.827054 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:45.827055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:45.827055 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:45.827312 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:45.827576 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:45.827577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:45.827578 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:45.827668 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:45.827997 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:45.827998 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:45.828043 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 14:57:45.828113 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:57:45.828196 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 14:57:45.828321 10884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.820861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53368 (local address 127.20.116.189:42873)
0928 14:57:45.820926 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:45.820928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:45.820948 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:57:45.821000 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:45.821002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:45.821012 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:45.821065 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:45.821485 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.822004 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.823042 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.823331 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.823388 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.825415 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:45.825418 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:45.825418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:45.825428 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:45.826964 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.827380 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.827381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.827382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.827446 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.827729 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.827730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.827730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.827886 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:45.827968 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:45.828159 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 14:57:45.828242 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:57:45.828881 10804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:57:45.829268 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.829157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:43173 (local address 127.0.0.1:34734)
0928 14:57:45.829209 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.829236 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 14:57:45.829586 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.829532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:43173 (local address 127.0.0.1:34750)
0928 14:57:45.829555 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.829574 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260928 14:57:45.829747 10804 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.188:43173: Network error: Client connection negotiation failed: client connection to 127.20.116.188:43173: connect: Connection refused (error 111)
I20260928 14:57:45.860901 10804 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } attempt: 1
I20260928 14:57:45.861402 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.861240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:43173 (local address 127.0.0.1:34754)
0928 14:57:45.861309 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.861361 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 14:57:45.861858 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.861749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:43173 (local address 127.0.0.1:34762)
0928 14:57:45.861802 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.861835 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 14:57:45.861968 10804 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.188:43173: Network error: Client connection negotiation failed: client connection to 127.20.116.188:43173: connect: Connection refused (error 111)
W20260928 14:57:45.868789 10886 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:57:45.869000 10886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:45.869019 10886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:45.869040 10886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:45.870157 10886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:45.870191 10886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:45.870205 10886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:45.870218 10886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:45.870230 10886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:45.871712 10886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:39513,127.20.116.189:42873,127.20.116.188:43173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.188.keytab
--principal=kudu/127.20.116.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.188:43173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.116.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:45.871940 10886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:45.872188 10886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:45.877167 10894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:45.877157 10891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:45.877154 10892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:45.877285 10886 server_base.cc:1034] running on GCE node
I20260928 14:57:45.877642 10886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:45.877942 10886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:45.879119 10886 hybrid_clock.cc:648] HybridClock initialized: now 1790607465879103 us; error 28 us; skew 500 ppm
Sep 28 14:57:45 dist-test-slave-pxl6 krb5kdc[10723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607465, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:45.881770 10886 init.cc:377] Logged in from keytab as kudu/127.20.116.188@KRBTEST.COM (short username kudu)
I20260928 14:57:45.882215 10886 webserver.cc:460] Webserver started at http://127.20.116.188:40947/ using document root <none> and password file <none>
I20260928 14:57:45.882422 10886 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:45.882467 10886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:45.882634 10886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:45.883777 10886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-2/data/instance:
uuid: "a88f8a07d4b14f018d508043add56f29"
format_stamp: "Formatted at 2026-09-28 14:57:45 on dist-test-slave-pxl6"
server_key: "2ac9ea61e294394c05dcdd49ad1f916e"
server_key_iv: "38ffb0620fee275421bf25ffa082b643"
server_key_version: "encryptionkey@0"
I20260928 14:57:45.884101 10886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-2/wal/instance:
uuid: "a88f8a07d4b14f018d508043add56f29"
format_stamp: "Formatted at 2026-09-28 14:57:45 on dist-test-slave-pxl6"
server_key: "2ac9ea61e294394c05dcdd49ad1f916e"
server_key_iv: "38ffb0620fee275421bf25ffa082b643"
server_key_version: "encryptionkey@0"
I20260928 14:57:45.885483 10886 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:45.886238 10901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:45.886459 10886 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:57:45.886597 10886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-2/wal
uuid: "a88f8a07d4b14f018d508043add56f29"
format_stamp: "Formatted at 2026-09-28 14:57:45 on dist-test-slave-pxl6"
server_key: "2ac9ea61e294394c05dcdd49ad1f916e"
server_key_iv: "38ffb0620fee275421bf25ffa082b643"
server_key_version: "encryptionkey@0"
I20260928 14:57:45.886693 10886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:45.895723 10879 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } attempt: 2
I20260928 14:57:45.896224 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.896070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:43173 (local address 127.0.0.1:34770)
0928 14:57:45.896142 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.896185 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 14:57:45.896646 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.896493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:43173 (local address 127.0.0.1:34774)
0928 14:57:45.896542 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.896583 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 14:57:45.896847 10879 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.188:43173: Network error: Client connection negotiation failed: client connection to 127.20.116.188:43173: connect: Connection refused (error 111)
I20260928 14:57:45.908517 10886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:45.911275 10886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:45.911465 10886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:45.916006 10886 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.188:43173
I20260928 14:57:45.916042 10953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.188:43173 every 8 connection(s)
I20260928 14:57:45.916486 10886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-2/data/info.pb
I20260928 14:57:45.917213 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 10886
I20260928 14:57:45.917330 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607192794321-20946-0/minicluster-data/master-2/wal/instance
I20260928 14:57:45.917536 20946 external_mini_cluster.cc:1428] Setting key 00e3c04bc8be13662ff6f7638735bb44
I20260928 14:57:45.917662 10954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:45.920440 10954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39513 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:57:45 dist-test-slave-pxl6 krb5kdc[10723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.188@KRBTEST.COM
Sep 28 14:57:45 dist-test-slave-pxl6 krb5kdc[10723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607465, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.188@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:57:45.929042 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.921156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.20.116.190:39513)
0928 14:57:45.921224 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:45.921227 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:45.921368 (+ 141us) server_negotiation.cc:407] Connection header received
0928 14:57:45.921433 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:45.921434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:45.921443 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:45.921494 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:45.922105 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.922758 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.923870 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.924053 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.924108 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.926276 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:45.926277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:45.926278 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:45.926286 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:45.927647 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.928084 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.928086 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.928086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.928149 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.928395 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.928396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.928396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.928511 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:45.928589 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:45.928850 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 14:57:45.928941 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 14:57:45.929294 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.918168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34784 (local address 127.20.116.188:43173)
0928 14:57:45.918374 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:45.918383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:45.918412 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:57:45.919031 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:45.919038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:45.919094 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:45.919218 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:45.920036 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.921189 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.922383 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.923000 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.923077 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.925825 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:45.925834 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:45.925834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:45.925842 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:45.927760 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.928180 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.928183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.928184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.928282 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.928517 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.928518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.928518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.928734 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:45.928884 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:45.928986 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 14:57:45.929070 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":102,"threads_started":1}
I20260928 14:57:45.929293 10958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.921062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:39513 (local address 127.0.0.1:38244)
0928 14:57:45.921251 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.921297 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:45.921370 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:45.921539 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:45.921543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:45.921929 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:45.922017 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:45.922019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.922861 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.922862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:45.923778 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:45.923781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.924146 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.924148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:45.924248 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.924707 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:45.924718 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:45.926182 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:45.927771 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:45.927779 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:45.927781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:45.927991 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:45.928220 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:45.928221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:45.928222 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:45.928319 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:45.928581 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:45.928588 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:45.928721 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 14:57:45.928826 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 14:57:45.928980 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":102,"threads_started":1}
I20260928 14:57:45.929903 10954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:57:45 dist-test-slave-pxl6 krb5kdc[10723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607465, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.188@KRBTEST.COM for kudu/127.20.116.189@KRBTEST.COM
Sep 28 14:57:45 dist-test-slave-pxl6 krb5kdc[10723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:57:45.937114 10804 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } attempt: 2
I20260928 14:57:45.938192 10884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.930300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.20.116.189:42873)
0928 14:57:45.930356 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:45.930358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:45.930386 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:57:45.930463 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:45.930464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:45.930473 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:45.930520 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:45.930895 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.931633 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.932795 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.933079 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.933149 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.935131 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:45.935132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:45.935132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:45.935141 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:45.936814 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.937210 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.937211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.937212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.937281 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.937502 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.937503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.937504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.937678 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:45.937757 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:45.937980 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 14:57:45.938071 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:57:45.938555 10958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.930135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:42873 (local address 127.0.0.1:53378)
0928 14:57:45.930344 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.930354 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:45.930398 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:45.930572 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:45.930573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:45.930741 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:45.930814 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:45.930816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.931743 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.931744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:45.932696 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:45.932699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.933155 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.933156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:45.933215 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.933600 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:45.933604 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:45.935038 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:45.936892 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:45.936893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:45.936894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:45.937105 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:45.937344 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:45.937345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:45.937346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:45.937417 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:45.937784 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:45.937785 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:45.937834 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 14:57:45.938284 (+ 450us) client_negotiation.cc:240] Negotiation successful
0928 14:57:45.938403 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260928 14:57:45.939211 10954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:57:45.939329 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.930231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38256 (local address 127.20.116.190:39513)
0928 14:57:45.930291 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:45.930292 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:45.930300 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:57:45.930322 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:45.930323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:45.930333 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:45.930395 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:45.931044 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.931791 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.932917 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.933208 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.933271 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.935773 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:45.935774 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:45.935775 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:45.935782 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:45.937833 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.938408 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.938410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.938410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.938470 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.938728 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.938729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.938730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.938875 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:45.939080 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:45.939127 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 14:57:45.939215 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 14:57:45 dist-test-slave-pxl6 krb5kdc[10723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607465, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for kudu/127.20.116.188@KRBTEST.COM
Sep 28 14:57:45 dist-test-slave-pxl6 krb5kdc[10723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.189@KRBTEST.COM
Sep 28 14:57:45 dist-test-slave-pxl6 krb5kdc[10723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607465, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.188@KRBTEST.COM for kudu/127.20.116.188@KRBTEST.COM
I20260928 14:57:45.946424 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.937560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34800 (local address 127.20.116.188:43173)
0928 14:57:45.937620 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:45.937622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:45.937629 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:57:45.937647 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:45.937648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:45.937658 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:45.937720 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:45.938061 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.938772 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.939940 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.940301 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.940379 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.943178 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:45.943179 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:45.943180 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:45.943188 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:45.945076 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.945537 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.945538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.945539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.945600 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.945840 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.945842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.945842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.945967 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:45.946046 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:45.946227 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 14:57:45.946314 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:57:45.946424 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.937388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:43173 (local address 127.0.0.1:34800)
0928 14:57:45.937437 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.937447 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:45.937483 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:45.937738 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:45.937739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:45.937936 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:45.937976 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:45.937978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.938917 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.938918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:45.939808 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:45.939811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.940393 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.940394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:45.940456 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.941430 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:45.941437 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:45.943069 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:45.945184 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:45.945185 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:45.945185 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:45.945440 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:45.945680 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:45.945681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:45.945681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:45.945745 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:45.946054 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:45.946055 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:45.946105 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 14:57:45.946199 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 14:57:45.946314 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 14:57:45.947970 10884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.939939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53388 (local address 127.20.116.189:42873)
0928 14:57:45.940001 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:45.940003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:45.940026 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:57:45.940087 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:45.940088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:45.940097 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:45.940146 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:45.940744 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.941295 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.941999 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.942216 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.942288 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.944916 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:45.944918 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:45.944918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:45.944926 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:45.946536 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.947056 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.947057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.947058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.947120 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.947346 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.947348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.947349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.947507 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:45.947718 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:45.947761 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 14:57:45.947844 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:57:45.948637 10804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d: Bootstrap starting.
I20260928 14:57:45.949307 10804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:45.949398 10958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.939537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:43173 (local address 127.0.0.1:34808)
0928 14:57:45.939731 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.939742 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:45.939783 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:45.940086 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:45.940087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:45.940240 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:45.940274 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:45.940275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.941451 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.941452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:45.942306 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:45.942309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.943198 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.943199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:45.943275 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.943899 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:45.943910 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:45.945441 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:45.948208 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:45.948208 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:45.948209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:45.948404 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:45.948631 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:45.948632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:45.948633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:45.948691 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:45.949026 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:45.949027 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:45.949071 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 14:57:45.949168 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 14:57:45.949275 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260928 14:57:45.949584 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.939605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34808 (local address 127.20.116.188:43173)
0928 14:57:45.939759 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:45.939761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:45.939928 (+ 167us) server_negotiation.cc:407] Connection header received
0928 14:57:45.939950 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:45.939951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:45.939963 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:45.940036 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:45.940353 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.941354 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.942398 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.942775 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.942848 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.945538 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:45.945539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:45.945540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:45.945546 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:45.948115 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.948493 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.948494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.948494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.948561 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.948798 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.948799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.948799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.948949 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:45.949034 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:45.949192 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 14:57:45.949335 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260928 14:57:45.949682 10804 log.cc:824] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:45.950742 10804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d: No bootstrap required, opened a new log
I20260928 14:57:45.951124 10954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a88f8a07d4b14f018d508043add56f29: Bootstrap starting.
I20260928 14:57:45.951372 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:57:45.951967 10954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a88f8a07d4b14f018d508043add56f29: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:45.952267 10954 log.cc:824] T 00000000000000000000000000000000 P a88f8a07d4b14f018d508043add56f29: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:45.952468 10778 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ed2839effc894d069a1dc67cb741b89d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38256
I20260928 14:57:45.952997 10804 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2839effc894d069a1dc67cb741b89d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39513 } } peers { permanent_uuid: "4254c5510cb340c09fe6e8c08be95bcf" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } } peers { permanent_uuid: "a88f8a07d4b14f018d508043add56f29" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } }
I20260928 14:57:45.953135 10804 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2839effc894d069a1dc67cb741b89d, State: Initialized, Role: FOLLOWER
I20260928 14:57:45.953179 10954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a88f8a07d4b14f018d508043add56f29: No bootstrap required, opened a new log
I20260928 14:57:45.953241 10804 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2839effc894d069a1dc67cb741b89d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39513 } } peers { permanent_uuid: "4254c5510cb340c09fe6e8c08be95bcf" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } } peers { permanent_uuid: "a88f8a07d4b14f018d508043add56f29" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } }
I20260928 14:57:45.953502 10962 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2839effc894d069a1dc67cb741b89d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39513 } } peers { permanent_uuid: "4254c5510cb340c09fe6e8c08be95bcf" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } } peers { permanent_uuid: "a88f8a07d4b14f018d508043add56f29" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } } }
I20260928 14:57:45.953577 10804 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:45.953614 10778 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 14:57:45.953727 10962 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:57:45.953727 10778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:45.954835 10778 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2839effc894d069a1dc67cb741b89d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39513 } } peers { permanent_uuid: "4254c5510cb340c09fe6e8c08be95bcf" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } } peers { permanent_uuid: "a88f8a07d4b14f018d508043add56f29" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } }
I20260928 14:57:45.955104 10778 leader_election.cc:288] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d [CANDIDATE]: Term 1 election: Requested vote from peers 4254c5510cb340c09fe6e8c08be95bcf (127.20.116.189:42873), a88f8a07d4b14f018d508043add56f29 (127.20.116.188:43173)
I20260928 14:57:45.955540 10954 raft_consensus.cc:348] T 00000000000000000000000000000000 P a88f8a07d4b14f018d508043add56f29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2839effc894d069a1dc67cb741b89d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39513 } } peers { permanent_uuid: "4254c5510cb340c09fe6e8c08be95bcf" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } } peers { permanent_uuid: "a88f8a07d4b14f018d508043add56f29" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } }
I20260928 14:57:45.955691 10954 raft_consensus.cc:729] T 00000000000000000000000000000000 P a88f8a07d4b14f018d508043add56f29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a88f8a07d4b14f018d508043add56f29, State: Initialized, Role: FOLLOWER
I20260928 14:57:45.955796 10954 consensus_queue.cc:260] T 00000000000000000000000000000000 P a88f8a07d4b14f018d508043add56f29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2839effc894d069a1dc67cb741b89d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39513 } } peers { permanent_uuid: "4254c5510cb340c09fe6e8c08be95bcf" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } } peers { permanent_uuid: "a88f8a07d4b14f018d508043add56f29" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } }
I20260928 14:57:45.956147 10954 sys_catalog.cc:564] T 00000000000000000000000000000000 P a88f8a07d4b14f018d508043add56f29 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:45.956110 10975 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88f8a07d4b14f018d508043add56f29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2839effc894d069a1dc67cb741b89d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39513 } } peers { permanent_uuid: "4254c5510cb340c09fe6e8c08be95bcf" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } } peers { permanent_uuid: "a88f8a07d4b14f018d508043add56f29" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } } }
I20260928 14:57:45.956383 10975 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88f8a07d4b14f018d508043add56f29 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:57:45.956684 10928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ed2839effc894d069a1dc67cb741b89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a88f8a07d4b14f018d508043add56f29"
I20260928 14:57:45.956812 10928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a88f8a07d4b14f018d508043add56f29 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:45.958313 10928 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a88f8a07d4b14f018d508043add56f29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2839effc894d069a1dc67cb741b89d in term 1.
W20260928 14:57:45.955590 10973 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:57:45.959170 10973 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:57:45.958745 10752 leader_election.cc:302] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a88f8a07d4b14f018d508043add56f29, ed2839effc894d069a1dc67cb741b89d; no voters:
I20260928 14:57:45.959406 10962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d [term 1 FOLLOWER]: Leader election won for term 1
W20260928 14:57:45.959470 10993 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a88f8a07d4b14f018d508043add56f29: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:57:45.959527 10993 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:57:45.959564 10962 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d [term 1 LEADER]: Becoming Leader. State: Replica: ed2839effc894d069a1dc67cb741b89d, State: Running, Role: LEADER
I20260928 14:57:45.959654 10962 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2839effc894d069a1dc67cb741b89d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39513 } } peers { permanent_uuid: "4254c5510cb340c09fe6e8c08be95bcf" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } } peers { permanent_uuid: "a88f8a07d4b14f018d508043add56f29" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } }
I20260928 14:57:45.960103 10962 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed2839effc894d069a1dc67cb741b89d. Latest consensus state: current_term: 1 leader_uuid: "ed2839effc894d069a1dc67cb741b89d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2839effc894d069a1dc67cb741b89d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39513 } } peers { permanent_uuid: "4254c5510cb340c09fe6e8c08be95bcf" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } } peers { permanent_uuid: "a88f8a07d4b14f018d508043add56f29" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } } }
I20260928 14:57:45.960189 10962 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:45.962462 10736 master_runner.cc:186] Error getting master registration for 127.20.116.189:42873: OK, instance_id { permanent_uuid: "4254c5510cb340c09fe6e8c08be95bcf" instance_seqno: 1790607465752462 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:57:45.964921 10736 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:57:45.962433 10886 master_runner.cc:186] Error getting master registration for 127.20.116.189:42873: OK, instance_id { permanent_uuid: "4254c5510cb340c09fe6e8c08be95bcf" instance_seqno: 1790607465752462 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:57:45.965255 10886 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:57:45.965281 10997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:45.965499 10997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:45.968505 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.956962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34820 (local address 127.20.116.188:43173)
0928 14:57:45.959344 (+ 2382us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:45.959347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:45.959356 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:57:45.959378 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:45.959379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:45.959389 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:45.959453 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:45.960091 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.960962 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.962636 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.962969 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.963063 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.964824 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:45.964826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:45.964826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:45.964834 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:45.966850 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.967384 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.967386 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.967387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.967454 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.967684 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.967684 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.967685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.967825 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:45.967964 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:45.968273 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 14:57:45.968359 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2361}
I20260928 14:57:45.971141 10928 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a88f8a07d4b14f018d508043add56f29 [term 1 FOLLOWER]: Refusing update from remote peer ed2839effc894d069a1dc67cb741b89d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:45.971380 10884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.958150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53404 (local address 127.20.116.189:42873)
0928 14:57:45.958507 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:45.958510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:45.959786 (+ 1276us) server_negotiation.cc:407] Connection header received
0928 14:57:45.959866 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:45.959868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:45.959878 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:45.959940 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:45.960629 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.961375 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.965843 (+ 4468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.966233 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.966292 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.968008 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:45.968009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:45.968009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:45.968017 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:45.969849 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.970316 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.970318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.970319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.970396 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.970668 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.970669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.970670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.970839 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:45.970952 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:45.971177 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 14:57:45.971264 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326}
I20260928 14:57:45.972636 10995 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a88f8a07d4b14f018d508043add56f29" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:45.974071 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.958666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38268 (local address 127.20.116.190:39513)
0928 14:57:45.958834 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:45.958837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:45.958846 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:57:45.958869 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:45.958870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:45.958881 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:45.958941 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:45.961286 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.961969 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.966831 (+ 4862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.967119 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.967163 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.969701 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:45.969703 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:45.969704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:45.969713 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:45.972400 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.972861 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.972863 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.972863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.972936 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.973169 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.973170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.973170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.973304 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:45.973414 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:45.973883 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 14:57:45.973964 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260928 14:57:45.974436 10975 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88f8a07d4b14f018d508043add56f29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed2839effc894d069a1dc67cb741b89d. Latest consensus state: current_term: 1 leader_uuid: "ed2839effc894d069a1dc67cb741b89d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2839effc894d069a1dc67cb741b89d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39513 } } peers { permanent_uuid: "4254c5510cb340c09fe6e8c08be95bcf" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } } peers { permanent_uuid: "a88f8a07d4b14f018d508043add56f29" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } } }
I20260928 14:57:45.974555 10975 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88f8a07d4b14f018d508043add56f29 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:57:45.975530 10962 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed2839effc894d069a1dc67cb741b89d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2839effc894d069a1dc67cb741b89d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39513 } } peers { permanent_uuid: "4254c5510cb340c09fe6e8c08be95bcf" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } } peers { permanent_uuid: "a88f8a07d4b14f018d508043add56f29" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } } }
I20260928 14:57:45.975634 10962 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:45.975752 10975 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88f8a07d4b14f018d508043add56f29 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed2839effc894d069a1dc67cb741b89d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2839effc894d069a1dc67cb741b89d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39513 } } peers { permanent_uuid: "4254c5510cb340c09fe6e8c08be95bcf" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } } peers { permanent_uuid: "a88f8a07d4b14f018d508043add56f29" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } } }
I20260928 14:57:45.975844 10975 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88f8a07d4b14f018d508043add56f29 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:57:45.976382 10999 mvcc.cc:205] Tried to move back new op lower bound from 7334328180613025792 to 7334328180571602944. Current Snapshot: MvccSnapshot[applied={T|T < 7334328180613025792}]
I20260928 14:57:45.977023 10997 catalog_manager.cc:1347] Generated new cluster ID: 333422f1bd82415b93b773c2c497b1e8
I20260928 14:57:45.977077 10997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:45.985320 10997 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:45.986143 10997 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:45.987986 10879 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } attempt: 3
Sep 28 14:57:45 dist-test-slave-pxl6 krb5kdc[10723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607465, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.189@KRBTEST.COM for kudu/127.20.116.188@KRBTEST.COM
I20260928 14:57:45.996850 10997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d: Generated new TSK 0
I20260928 14:57:45.997450 10997 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:57:45.998013 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.988344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34832 (local address 127.20.116.188:43173)
0928 14:57:45.988387 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:45.988389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:45.988397 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:57:45.988481 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:45.988482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:45.988494 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:45.988553 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:45.988895 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.989762 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.990831 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.991178 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.991256 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.993610 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:45.993612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:45.993613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:45.993621 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:45.996528 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.997008 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.997010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.997010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.997074 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:45.997334 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:45.997334 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:45.997335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:45.997478 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:45.997548 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:45.997832 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 14:57:45.997915 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:57:45.998013 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:45.988297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:43173 (local address 127.0.0.1:34832)
0928 14:57:45.988353 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:45.988367 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:45.988413 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:45.988568 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:45.988568 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:45.988780 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:45.988829 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:45.988831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.989841 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.989843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:45.990695 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:45.990698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:45.991259 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:45.991260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:45.991337 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:45.991854 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:45.991874 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:45.993516 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:45.996656 (+ 3140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:45.996657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:45.996658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:45.996925 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:45.997155 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:45.997156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:45.997156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:45.997250 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:45.997604 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:45.997605 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:45.997674 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 14:57:45.997787 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 14:57:45.997904 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 14:57:45.999840 10879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4254c5510cb340c09fe6e8c08be95bcf: Bootstrap starting.
I20260928 14:57:46.000664 10879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4254c5510cb340c09fe6e8c08be95bcf: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:46.001071 10879 log.cc:824] T 00000000000000000000000000000000 P 4254c5510cb340c09fe6e8c08be95bcf: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:46.001992 10879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4254c5510cb340c09fe6e8c08be95bcf: No bootstrap required, opened a new log
I20260928 14:57:46.004484 10879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4254c5510cb340c09fe6e8c08be95bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2839effc894d069a1dc67cb741b89d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39513 } } peers { permanent_uuid: "4254c5510cb340c09fe6e8c08be95bcf" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } } peers { permanent_uuid: "a88f8a07d4b14f018d508043add56f29" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } }
I20260928 14:57:46.004637 10879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4254c5510cb340c09fe6e8c08be95bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4254c5510cb340c09fe6e8c08be95bcf, State: Initialized, Role: FOLLOWER
I20260928 14:57:46.004726 10879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4254c5510cb340c09fe6e8c08be95bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2839effc894d069a1dc67cb741b89d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39513 } } peers { permanent_uuid: "4254c5510cb340c09fe6e8c08be95bcf" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } } peers { permanent_uuid: "a88f8a07d4b14f018d508043add56f29" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } }
I20260928 14:57:46.005108 10879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4254c5510cb340c09fe6e8c08be95bcf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:46.005453 10852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4254c5510cb340c09fe6e8c08be95bcf [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:46.006556 10852 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4254c5510cb340c09fe6e8c08be95bcf [term 1 FOLLOWER]: Refusing update from remote peer ed2839effc894d069a1dc67cb741b89d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:46.006717 10853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ed2839effc894d069a1dc67cb741b89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4254c5510cb340c09fe6e8c08be95bcf"
I20260928 14:57:46.006991 11004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4254c5510cb340c09fe6e8c08be95bcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2839effc894d069a1dc67cb741b89d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39513 } } peers { permanent_uuid: "4254c5510cb340c09fe6e8c08be95bcf" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } } peers { permanent_uuid: "a88f8a07d4b14f018d508043add56f29" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } } }
I20260928 14:57:46.007131 11004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4254c5510cb340c09fe6e8c08be95bcf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:57:46.007793 10962 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4254c5510cb340c09fe6e8c08be95bcf" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:46.007944 10853 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4254c5510cb340c09fe6e8c08be95bcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2839effc894d069a1dc67cb741b89d in term 1.
I20260928 14:57:46.009536 11004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4254c5510cb340c09fe6e8c08be95bcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed2839effc894d069a1dc67cb741b89d. Latest consensus state: current_term: 1 leader_uuid: "ed2839effc894d069a1dc67cb741b89d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2839effc894d069a1dc67cb741b89d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39513 } } peers { permanent_uuid: "4254c5510cb340c09fe6e8c08be95bcf" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } } peers { permanent_uuid: "a88f8a07d4b14f018d508043add56f29" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } } }
I20260928 14:57:46.009620 11004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4254c5510cb340c09fe6e8c08be95bcf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:57:46.009660 11004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4254c5510cb340c09fe6e8c08be95bcf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed2839effc894d069a1dc67cb741b89d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2839effc894d069a1dc67cb741b89d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39513 } } peers { permanent_uuid: "4254c5510cb340c09fe6e8c08be95bcf" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } } peers { permanent_uuid: "a88f8a07d4b14f018d508043add56f29" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } } }
I20260928 14:57:46.009707 11004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4254c5510cb340c09fe6e8c08be95bcf [sys.catalog]: This master's current role is: FOLLOWER
W20260928 14:57:46.015159 11016 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4254c5510cb340c09fe6e8c08be95bcf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:57:46.015228 11016 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:57:46.015884 11017 mvcc.cc:205] Tried to move back new op lower bound from 7334328180708212736 to 7334328180571602944. Current Snapshot: MvccSnapshot[applied={T|T < 7334328180613025792}]
W20260928 14:57:46.034853 10807 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:46.033162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.20.116.190:39513)
0928 14:57:46.033211 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.033213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.034136 (+ 923us) server_negotiation.cc:407] Connection header received
0928 14:57:46.034246 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.034247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.034260 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.034292 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:46.034778 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38274: BlockingRecv error: recv got EOF from 127.0.0.1:38274 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 14:57:46.035607 10884 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:46.033552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53406 (local address 127.20.116.189:42873)
0928 14:57:46.033595 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.033596 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.035103 (+ 1507us) server_negotiation.cc:407] Connection header received
0928 14:57:46.035200 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.035201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.035212 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.035241 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:46.035544 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53406: BlockingRecv error: recv got EOF from 127.0.0.1:53406 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 14:57:46.036432 10959 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:46.033877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34846 (local address 127.20.116.188:43173)
0928 14:57:46.033938 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.033941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.035843 (+ 1902us) server_negotiation.cc:407] Connection header received
0928 14:57:46.035939 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.035940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.035951 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.035980 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:46.036358 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34846: BlockingRecv error: recv got EOF from 127.0.0.1:34846 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 14:57:46.038888 10884 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:46.038095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.20.116.189:42873)
0928 14:57:46.038129 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.038130 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.038326 (+ 196us) server_negotiation.cc:407] Connection header received
0928 14:57:46.038427 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.038428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.038445 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.038473 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:46.038827 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53414: BlockingRecv error: recv got EOF from 127.0.0.1:53414 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 14:57:46.041785 10884 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:46.041060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53426 (local address 127.20.116.189:42873)
0928 14:57:46.041091 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.041093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.041306 (+ 213us) server_negotiation.cc:407] Connection header received
0928 14:57:46.041396 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.041397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.041406 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.041433 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:46.041744 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53426: BlockingRecv error: recv got EOF from 127.0.0.1:53426 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 14:57:46.045589 10884 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:46.044876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53442 (local address 127.20.116.189:42873)
0928 14:57:46.044908 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.044910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.045127 (+ 217us) server_negotiation.cc:407] Connection header received
0928 14:57:46.045218 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.045218 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.045228 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.045255 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:46.045547 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53442: BlockingRecv error: recv got EOF from 127.0.0.1:53442 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 14:57:46.051462 10884 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:46.050747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53448 (local address 127.20.116.189:42873)
0928 14:57:46.050779 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.050781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.051006 (+ 225us) server_negotiation.cc:407] Connection header received
0928 14:57:46.051092 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.051093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.051102 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.051128 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:46.051420 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53448: BlockingRecv error: recv got EOF from 127.0.0.1:53448 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 14:57:46.062018 10884 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:46.060714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53452 (local address 127.20.116.189:42873)
0928 14:57:46.060758 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.060760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.061016 (+ 256us) server_negotiation.cc:407] Connection header received
0928 14:57:46.061116 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.061118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.061129 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.061222 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:46.061959 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53452: BlockingRecv error: recv got EOF from 127.0.0.1:53452 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 14:57:46.080449 10884 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:46.079471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53468 (local address 127.20.116.189:42873)
0928 14:57:46.079515 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.079518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.079812 (+ 294us) server_negotiation.cc:407] Connection header received
0928 14:57:46.079923 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.079924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.079937 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.079967 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:46.080388 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53468: BlockingRecv error: recv got EOF from 127.0.0.1:53468 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 14:57:46.114892 10884 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:46.113862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53470 (local address 127.20.116.189:42873)
0928 14:57:46.113913 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.113916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.114206 (+ 290us) server_negotiation.cc:407] Connection header received
0928 14:57:46.114328 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.114330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.114343 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.114377 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:46.114829 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53470: BlockingRecv error: recv got EOF from 127.0.0.1:53470 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 14:57:46.181159 10884 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:46.180297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53474 (local address 127.20.116.189:42873)
0928 14:57:46.180350 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.180353 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.180509 (+ 156us) server_negotiation.cc:407] Connection header received
0928 14:57:46.180569 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.180570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.180584 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.180663 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:46.181097 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53474: BlockingRecv error: recv got EOF from 127.0.0.1:53474 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 14:57:46.311170 10884 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:46.310422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53488 (local address 127.20.116.189:42873)
0928 14:57:46.310492 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.310494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.310639 (+ 145us) server_negotiation.cc:407] Connection header received
0928 14:57:46.310687 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.310688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.310700 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.310777 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:46.311094 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53488: BlockingRecv error: recv got EOF from 127.0.0.1:53488 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:57:46.428092 10962 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed2839effc894d069a1dc67cb741b89d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2839effc894d069a1dc67cb741b89d" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 39513 } } peers { permanent_uuid: "4254c5510cb340c09fe6e8c08be95bcf" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 42873 } } peers { permanent_uuid: "a88f8a07d4b14f018d508043add56f29" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 43173 } } }
I20260928 14:57:46.428257 10962 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2839effc894d069a1dc67cb741b89d [sys.catalog]: This master's current role is: LEADER
W20260928 14:57:46.569140 10884 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:46.568531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53502 (local address 127.20.116.189:42873)
0928 14:57:46.568604 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:46.568606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:46.568615 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:57:46.568680 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:46.568681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:46.568692 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:46.568759 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:46.569083 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53502: BlockingRecv error: recv got EOF from 127.0.0.1:53502 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:57:46.960054 10993 catalog_manager.cc:1259] Loaded cluster ID: 333422f1bd82415b93b773c2c497b1e8
I20260928 14:57:46.960122 10993 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a88f8a07d4b14f018d508043add56f29: loading cluster ID for follower catalog manager: success
I20260928 14:57:46.960973 10993 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a88f8a07d4b14f018d508043add56f29: acquiring CA information for follower catalog manager: success
I20260928 14:57:46.961289 10993 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a88f8a07d4b14f018d508043add56f29: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:57:47.015729 11016 catalog_manager.cc:1259] Loaded cluster ID: 333422f1bd82415b93b773c2c497b1e8
I20260928 14:57:47.015812 11016 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4254c5510cb340c09fe6e8c08be95bcf: loading cluster ID for follower catalog manager: success
I20260928 14:57:47.016688 11016 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4254c5510cb340c09fe6e8c08be95bcf: acquiring CA information for follower catalog manager: success
I20260928 14:57:47.017019 11016 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4254c5510cb340c09fe6e8c08be95bcf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:57:47.085475 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:47.082583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53506 (local address 127.20.116.189:42873)
0928 14:57:47.082718 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:47.082720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:47.082755 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:57:47.082829 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:47.082830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:47.082886 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:47.082965 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:47.083181 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:47.083765 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:47.084641 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:47.084852 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:47.084908 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:47.085136 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:47.085250 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:47.085313 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:47.085387 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 14:57:47.085404 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260928 14:57:47.090202 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:47.087587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34848 (local address 127.20.116.188:43173)
0928 14:57:47.087759 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:47.087761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:47.087769 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:57:47.087814 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:47.087815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:47.087851 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:47.087944 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:47.088074 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:47.088569 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:47.089371 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:47.089586 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:47.089664 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:47.089833 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:47.089965 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:47.090054 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:47.090077 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 14:57:47.090101 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260928 14:57:47.094822 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:47.092196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34864 (local address 127.20.116.188:43173)
0928 14:57:47.092251 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:47.092253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:47.092260 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:57:47.092356 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:47.092357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:47.092367 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:47.092398 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:47.092612 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:47.093323 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:47.094070 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:47.094282 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:47.094365 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:47.094550 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:47.094584 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:47.094639 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:47.094701 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 14:57:47.094728 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:57:47.095239 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:47.092128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53514 (local address 127.20.116.189:42873)
0928 14:57:47.092170 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:47.092172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:47.092275 (+ 103us) server_negotiation.cc:407] Connection header received
0928 14:57:47.092335 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:47.092336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:47.092350 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:47.092398 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:47.092553 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:47.093303 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:47.094388 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:47.094711 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:47.094761 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:47.094971 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:47.095002 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:47.095030 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:47.095125 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 14:57:47.095158 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:57:47.095516 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:47.091972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38278 (local address 127.20.116.190:39513)
0928 14:57:47.092082 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:47.092084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:47.092196 (+ 112us) server_negotiation.cc:407] Connection header received
0928 14:57:47.092412 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:47.092414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:47.092471 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:47.092551 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:47.092741 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:47.093241 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:47.094381 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:47.094715 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:47.094772 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:47.095002 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:47.095115 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:47.095169 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:47.095412 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 14:57:47.095437 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":56,"threads_started":1}
I20260928 14:57:47.097180 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 10736
I20260928 14:57:47.102844 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 10811
I20260928 14:57:47.108157 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 10886
2026-09-28T14:57:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2804 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:47 dist-test-slave-pxl6 krb5kdc[11105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:47 dist-test-slave-pxl6 krb5kdc[11105](info): set up 2 sockets
Sep 28 14:57:47 dist-test-slave-pxl6 krb5kdc[11105](info): commencing operation
krb5kdc: starting...
W20260928 14:57:48.155791 20946 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/test-user.keytab.
Sep 28 14:57:48 dist-test-slave-pxl6 krb5kdc[11105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
Entry for principal kudu/127.20.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab.
I20260928 14:57:48.279526 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:36537
--webserver_interface=127.20.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:43279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:36537,127.20.116.189:40935,127.20.116.188:37339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:48.363214 11118 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:57:48.363468 11118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:48.363499 11118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:48.363543 11118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:48.364728 11118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:48.364796 11118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:48.364820 11118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:48.364841 11118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:48.364871 11118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:48.366360 11118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:43279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:36537,127.20.116.189:40935,127.20.116.188:37339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.190.keytab
--principal=kudu/127.20.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.190:36537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:48.366696 11118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:48.366942 11118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:48.372073 11123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:48.372088 11124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:48.372138 11118 server_base.cc:1034] running on GCE node
W20260928 14:57:48.372082 11126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:48.372555 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:48.372859 11118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:48.374039 11118 hybrid_clock.cc:648] HybridClock initialized: now 1790607468374026 us; error 46 us; skew 500 ppm
Sep 28 14:57:48 dist-test-slave-pxl6 krb5kdc[11105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607468, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:48.376628 11118 init.cc:377] Logged in from keytab as kudu/127.20.116.190@KRBTEST.COM (short username kudu)
I20260928 14:57:48.377100 11118 webserver.cc:460] Webserver started at http://127.20.116.190:46665/ using document root <none> and password file <none>
I20260928 14:57:48.377342 11118 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:48.377411 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:48.377555 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:48.378738 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-0/data/instance:
uuid: "88c5235003654094ae04e578c54cc32b"
format_stamp: "Formatted at 2026-09-28 14:57:48 on dist-test-slave-pxl6"
server_key: "3e7bf436ee7a4d90a70a40f93f259032"
server_key_iv: "3570398d8ee458282bb879c9469107a4"
server_key_version: "encryptionkey@0"
I20260928 14:57:48.379103 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-0/wal/instance:
uuid: "88c5235003654094ae04e578c54cc32b"
format_stamp: "Formatted at 2026-09-28 14:57:48 on dist-test-slave-pxl6"
server_key: "3e7bf436ee7a4d90a70a40f93f259032"
server_key_iv: "3570398d8ee458282bb879c9469107a4"
server_key_version: "encryptionkey@0"
I20260928 14:57:48.380499 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:57:48.381347 11133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:48.381680 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:57:48.381814 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-0/wal
uuid: "88c5235003654094ae04e578c54cc32b"
format_stamp: "Formatted at 2026-09-28 14:57:48 on dist-test-slave-pxl6"
server_key: "3e7bf436ee7a4d90a70a40f93f259032"
server_key_iv: "3570398d8ee458282bb879c9469107a4"
server_key_version: "encryptionkey@0"
I20260928 14:57:48.381896 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:48.393489 11118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:48.395769 11118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:48.395938 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:48.400270 11118 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.190:36537
I20260928 14:57:48.400306 11185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.190:36537 every 8 connection(s)
I20260928 14:57:48.400687 11118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:48.401679 11186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:48.403425 11186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36537 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:57:48.404294 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 11118
I20260928 14:57:48.404392 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-0/wal/instance
I20260928 14:57:48.404611 20946 external_mini_cluster.cc:1428] Setting key 1451de1cc45067ba8d206ad3150fba18
Sep 28 14:57:48 dist-test-slave-pxl6 krb5kdc[11105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607468, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.116.189@KRBTEST.COM; defaulting to no policy
I20260928 14:57:48.414978 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.404395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:36537 (local address 127.0.0.1:53306)
0928 14:57:48.404688 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.404746 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:48.405290 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:48.405609 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:48.405612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:48.405926 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:48.406088 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:48.406093 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.407449 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.407451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:48.408305 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:48.408307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.409031 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.409032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:48.409095 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.409647 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:48.409656 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:48.411091 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:48.413073 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:48.413080 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:48.413082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:48.413285 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:48.413556 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:48.413558 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:48.413558 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:48.413681 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:48.414252 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:48.414276 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:48.414338 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 14:57:48.414426 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 14:57:48.414757 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":98,"threads_started":1}
I20260928 14:57:48.414985 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.404628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53306 (local address 127.20.116.190:36537)
0928 14:57:48.404746 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.404756 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.404839 (+ 83us) server_negotiation.cc:407] Connection header received
0928 14:57:48.405471 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.405473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.405500 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.405603 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.406215 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.407325 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.408399 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.408933 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.409061 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.411183 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:48.411193 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:48.411193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:48.411203 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:48.412999 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.413361 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.413364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.413365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.413483 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.413922 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.413924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.413924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.414173 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:48.414247 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:48.414475 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 14:57:48.414824 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":234,"server-negotiator.queue_time_us":85,"thread_start_us":63,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.189@KRBTEST.COM" created.
I20260928 14:57:48.415807 11186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:57:48.416218 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.416062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:40935 (local address 127.0.0.1:57790)
0928 14:57:48.416138 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.416190 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.189:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 14:57:48.416363 11137 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.116.189:40935: connect: Connection refused (error 111)
I20260928 14:57:48.416857 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.416738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:40935 (local address 127.0.0.1:57792)
0928 14:57:48.416806 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.416835 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.189:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 14:57:48.416994 11186 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.189:40935: Network error: Client connection negotiation failed: client connection to 127.20.116.189:40935: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.189.keytab.
Entry for principal kudu/127.20.116.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.189.keytab.
I20260928 14:57:48.430994 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.189:40935
--webserver_interface=127.20.116.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.189.keytab
--principal=kudu/127.20.116.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:43279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:36537,127.20.116.189:40935,127.20.116.188:37339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:57:48.466265 11186 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } attempt: 1
I20260928 14:57:48.466820 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.466675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:40935 (local address 127.0.0.1:57800)
0928 14:57:48.466745 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.466787 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.189:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 14:57:48.467190 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.467089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:40935 (local address 127.0.0.1:57802)
0928 14:57:48.467138 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.467165 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.189:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 14:57:48.467291 11186 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.189:40935: Network error: Client connection negotiation failed: client connection to 127.20.116.189:40935: connect: Connection refused (error 111)
W20260928 14:57:48.514719 11193 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:57:48.514967 11193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:48.514994 11193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:48.515033 11193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:48.516279 11193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:48.516347 11193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:48.516371 11193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:48.516399 11193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:48.516431 11193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:48.517956 11193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:43279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:36537,127.20.116.189:40935,127.20.116.188:37339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.189.keytab
--principal=kudu/127.20.116.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.189:40935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.116.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:48.518239 11193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:48.518468 11193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:48.523504 11201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:48.523574 11193 server_base.cc:1034] running on GCE node
W20260928 14:57:48.523514 11199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:48.523514 11198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:48.524024 11193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:48.524328 11193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:48.525473 11193 hybrid_clock.cc:648] HybridClock initialized: now 1790607468525460 us; error 29 us; skew 500 ppm
Sep 28 14:57:48 dist-test-slave-pxl6 krb5kdc[11105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607468, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:48.528038 11193 init.cc:377] Logged in from keytab as kudu/127.20.116.189@KRBTEST.COM (short username kudu)
I20260928 14:57:48.528424 11193 webserver.cc:460] Webserver started at http://127.20.116.189:46079/ using document root <none> and password file <none>
I20260928 14:57:48.528609 11193 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:48.528650 11193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:48.528755 11193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:48.529855 11193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-1/data/instance:
uuid: "6eb63e4ccd174039bbfe8572744db2ab"
format_stamp: "Formatted at 2026-09-28 14:57:48 on dist-test-slave-pxl6"
server_key: "d5836ee93cc2f1455663bee3b727b873"
server_key_iv: "73f811f927c10c8361e2ef8ece3ad670"
server_key_version: "encryptionkey@0"
I20260928 14:57:48.530164 11193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-1/wal/instance:
uuid: "6eb63e4ccd174039bbfe8572744db2ab"
format_stamp: "Formatted at 2026-09-28 14:57:48 on dist-test-slave-pxl6"
server_key: "d5836ee93cc2f1455663bee3b727b873"
server_key_iv: "73f811f927c10c8361e2ef8ece3ad670"
server_key_version: "encryptionkey@0"
I20260928 14:57:48.531579 11193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:48.532383 11208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:48.532668 11193 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:57:48.532778 11193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-1/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-1/wal
uuid: "6eb63e4ccd174039bbfe8572744db2ab"
format_stamp: "Formatted at 2026-09-28 14:57:48 on dist-test-slave-pxl6"
server_key: "d5836ee93cc2f1455663bee3b727b873"
server_key_iv: "73f811f927c10c8361e2ef8ece3ad670"
server_key_version: "encryptionkey@0"
I20260928 14:57:48.532862 11193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:48.535439 11186 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } attempt: 2
I20260928 14:57:48.535998 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.535849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:40935 (local address 127.0.0.1:57804)
0928 14:57:48.535918 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.535961 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.189:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 14:57:48.536409 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.536308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:40935 (local address 127.0.0.1:57816)
0928 14:57:48.536353 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.536385 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.189:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 14:57:48.536527 11186 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.189:40935: Network error: Client connection negotiation failed: client connection to 127.20.116.189:40935: connect: Connection refused (error 111)
I20260928 14:57:48.543800 11193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:48.546147 11193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:48.546303 11193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:48.550143 11193 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.189:40935
I20260928 14:57:48.550176 11260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.189:40935 every 8 connection(s)
I20260928 14:57:48.550572 11193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-1/data/info.pb
I20260928 14:57:48.551520 11261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:48.553099 11261 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36537 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:57:48.556200 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 11193
I20260928 14:57:48.556337 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-1/wal/instance
I20260928 14:57:48.556526 20946 external_mini_cluster.cc:1428] Setting key ffa944c316e8db6f7c4994c99d0d9259
Sep 28 14:57:48 dist-test-slave-pxl6 krb5kdc[11105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607468, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.189@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:57:48.562987 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.553872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53318 (local address 127.20.116.190:36537)
0928 14:57:48.553921 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.553923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.554130 (+ 207us) server_negotiation.cc:407] Connection header received
0928 14:57:48.554641 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.554641 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.554654 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.554720 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.555343 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.555822 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.557040 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.557392 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.557470 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.559992 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:48.559994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:48.559995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:48.560004 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:48.561457 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.561907 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.561908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.561908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.561974 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.562236 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.562237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.562238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.562361 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:48.562435 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:48.562798 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 14:57:48.562889 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:57:48.563241 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.553822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:36537 (local address 127.0.0.1:53318)
0928 14:57:48.554001 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.554047 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:48.554507 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:48.554787 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:48.554790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:48.555036 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:48.555265 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:48.555269 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.555905 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.555906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:48.556922 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:48.556925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.557503 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.557504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:48.557629 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.558334 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:48.558343 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:48.559878 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:48.561559 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:48.561567 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:48.561570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:48.561823 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:48.562046 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:48.562047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:48.562047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:48.562165 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:48.562439 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:48.562447 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:48.562664 (+ 217us) client_negotiation.cc:769] Sending connection context
0928 14:57:48.562775 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 14:57:48.562939 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260928 14:57:48.563804 11261 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.116.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.116.188@KRBTEST.COM" created.
Sep 28 14:57:48 dist-test-slave-pxl6 krb5kdc[11105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607468, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.189@KRBTEST.COM for kudu/127.20.116.189@KRBTEST.COM
I20260928 14:57:48.571481 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.564171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57824 (local address 127.20.116.189:40935)
0928 14:57:48.564328 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.564333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.564350 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:57:48.564369 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.564371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.564389 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.564477 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.564744 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.565431 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.566135 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.566462 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.566567 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.568677 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:48.568686 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:48.568686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:48.568694 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:48.570212 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.570608 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.570611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.570612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.570669 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.570868 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.570869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.570869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.571025 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:48.571130 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:48.571239 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 14:57:48.571344 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20260928 14:57:48.571489 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.564083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:40935 (local address 127.0.0.1:57824)
0928 14:57:48.564132 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.564143 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:48.564185 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:48.564477 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:48.564478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:48.564620 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:48.564666 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:48.564667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.565520 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.565521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:48.566033 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:48.566034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.566563 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.566564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:48.566630 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.567233 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:48.567239 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:48.568570 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:48.570323 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:48.570324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:48.570325 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:48.570490 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:48.570739 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:48.570740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:48.570740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:48.570795 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:48.571100 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:48.571100 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:48.571145 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 14:57:48.571211 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 14:57:48.571396 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":89}
I20260928 14:57:48.572309 11261 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:57:48.572713 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.572581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:37339 (local address 127.0.0.1:56044)
0928 14:57:48.572633 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.572681 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:37339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 14:57:48.572876 11211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.116.188:37339: connect: Connection refused (error 111)
I20260928 14:57:48.573309 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.573208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:37339 (local address 127.0.0.1:56054)
0928 14:57:48.573262 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.573286 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:37339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 14:57:48.573415 11261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.188:37339: Network error: Client connection negotiation failed: client connection to 127.20.116.188:37339: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.116.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.188.keytab.
Entry for principal kudu/127.20.116.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.188.keytab.
I20260928 14:57:48.582942 20946 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOCNdC/build/release/bin/kudu
/tmp/dist-test-taskHOCNdC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.188:37339
--webserver_interface=127.20.116.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.188.keytab
--principal=kudu/127.20.116.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.116.148:43279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.116.190:36537,127.20.116.189:40935,127.20.116.188:37339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:57:48.622673 11261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } attempt: 1
I20260928 14:57:48.623238 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.623080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:37339 (local address 127.0.0.1:56064)
0928 14:57:48.623158 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.623204 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:37339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 14:57:48.623682 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.623579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:37339 (local address 127.0.0.1:56068)
0928 14:57:48.623633 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.623660 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:37339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 14:57:48.623819 11261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.188:37339: Network error: Client connection negotiation failed: client connection to 127.20.116.188:37339: connect: Connection refused (error 111)
I20260928 14:57:48.627755 11186 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } attempt: 3
Sep 28 14:57:48 dist-test-slave-pxl6 krb5kdc[11105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607468, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for kudu/127.20.116.189@KRBTEST.COM
I20260928 14:57:48.635514 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.628139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:40935 (local address 127.0.0.1:57832)
0928 14:57:48.628225 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.628237 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:48.628317 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:48.628471 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:48.628472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:48.628619 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:48.628686 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:48.628688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.629617 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.629618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:48.630151 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:48.630152 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.630684 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.630685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:48.630758 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.631236 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:48.631240 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:48.632525 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:48.634330 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:48.634331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:48.634331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:48.634512 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:48.634831 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:48.634832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:48.634833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:48.634895 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:48.635191 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:48.635192 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:48.635250 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 14:57:48.635357 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 14:57:48.635441 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 14:57:48.635567 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.628176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57832 (local address 127.20.116.189:40935)
0928 14:57:48.628247 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.628249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.628302 (+ 53us) server_negotiation.cc:407] Connection header received
0928 14:57:48.628389 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.628390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.628401 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.628457 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.628761 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.629524 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.630242 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.630604 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.630671 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.632617 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:48.632619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:48.632620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:48.632628 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:48.634222 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.634664 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.634666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.634666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.634739 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.634960 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.634961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.634961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.635100 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:48.635179 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:48.635393 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 14:57:48.635475 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:57:48.636180 11186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:57:48.636552 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.636453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:37339 (local address 127.0.0.1:56082)
0928 14:57:48.636504 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.636533 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:37339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 14:57:48.636935 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.636820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:37339 (local address 127.0.0.1:56084)
0928 14:57:48.636873 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.636918 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:37339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 14:57:48.637110 11186 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.188:37339: Network error: Client connection negotiation failed: client connection to 127.20.116.188:37339: connect: Connection refused (error 111)
I20260928 14:57:48.668303 11186 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } attempt: 1
W20260928 14:57:48.668397 11268 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:57:48.668711 11268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:48.668751 11268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:48.668793 11268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 14:57:48.668808 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.668640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:37339 (local address 127.0.0.1:56090)
0928 14:57:48.668714 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.668755 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:37339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 14:57:48.669198 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.669097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:37339 (local address 127.0.0.1:56092)
0928 14:57:48.669143 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.669167 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:37339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 14:57:48.669289 11186 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.188:37339: Network error: Client connection negotiation failed: client connection to 127.20.116.188:37339: connect: Connection refused (error 111)
W20260928 14:57:48.670114 11268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:48.670153 11268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:48.670168 11268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:48.670181 11268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:48.670222 11268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:48.671725 11268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.116.148:43279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.116.190:36537,127.20.116.189:40935,127.20.116.188:37339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/krb5kdc/kudu_127.20.116.188.keytab
--principal=kudu/127.20.116.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.116.188:37339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.116.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:48.671988 11268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:48.672225 11268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:48.677417 11274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:48.677395 11273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:48.677462 11268 server_base.cc:1034] running on GCE node
W20260928 14:57:48.677395 11276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:48.677840 11268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:48.678138 11268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:48.679280 11268 hybrid_clock.cc:648] HybridClock initialized: now 1790607468679254 us; error 38 us; skew 500 ppm
Sep 28 14:57:48 dist-test-slave-pxl6 krb5kdc[11105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607468, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:48.681598 11268 init.cc:377] Logged in from keytab as kudu/127.20.116.188@KRBTEST.COM (short username kudu)
I20260928 14:57:48.682103 11268 webserver.cc:460] Webserver started at http://127.20.116.188:44629/ using document root <none> and password file <none>
I20260928 14:57:48.682337 11268 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:48.682404 11268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:48.682601 11268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:48.683950 11268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-2/data/instance:
uuid: "ce2fcbeb5e2b4457b0a6826010744455"
format_stamp: "Formatted at 2026-09-28 14:57:48 on dist-test-slave-pxl6"
server_key: "ec58dbe73ac231e943a83b2a380d1b23"
server_key_iv: "72e11598d4e0a36d722d7e26f45b394d"
server_key_version: "encryptionkey@0"
I20260928 14:57:48.684361 11268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-2/wal/instance:
uuid: "ce2fcbeb5e2b4457b0a6826010744455"
format_stamp: "Formatted at 2026-09-28 14:57:48 on dist-test-slave-pxl6"
server_key: "ec58dbe73ac231e943a83b2a380d1b23"
server_key_iv: "72e11598d4e0a36d722d7e26f45b394d"
server_key_version: "encryptionkey@0"
I20260928 14:57:48.685799 11268 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:48.686582 11283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:48.686790 11268 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:57:48.686890 11268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-2/data,/tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-2/wal
uuid: "ce2fcbeb5e2b4457b0a6826010744455"
format_stamp: "Formatted at 2026-09-28 14:57:48 on dist-test-slave-pxl6"
server_key: "ec58dbe73ac231e943a83b2a380d1b23"
server_key_iv: "72e11598d4e0a36d722d7e26f45b394d"
server_key_version: "encryptionkey@0"
I20260928 14:57:48.686976 11268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:48.691987 11261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } attempt: 2
I20260928 14:57:48.692453 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.692318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:37339 (local address 127.0.0.1:56096)
0928 14:57:48.692382 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.692422 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:37339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 14:57:48.692840 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.692738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:37339 (local address 127.0.0.1:56100)
0928 14:57:48.692786 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.692813 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.116.188:37339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 14:57:48.692927 11261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.116.188:37339: Network error: Client connection negotiation failed: client connection to 127.20.116.188:37339: connect: Connection refused (error 111)
I20260928 14:57:48.700538 11268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:48.702440 11268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:48.702634 11268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:48.706600 11268 rpc_server.cc:307] RPC server started. Bound to: 127.20.116.188:37339
I20260928 14:57:48.706640 11335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.116.188:37339 every 8 connection(s)
I20260928 14:57:48.707027 11268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-2/data/info.pb
I20260928 14:57:48.707998 20946 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOCNdC/build/release/bin/kudu as pid 11268
I20260928 14:57:48.708048 11336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:48.708151 20946 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOCNdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607192794321-20946-0/minicluster-data/master-2/wal/instance
I20260928 14:57:48.708379 20946 external_mini_cluster.cc:1428] Setting key c672f1cd10e81bc36982110012273109
I20260928 14:57:48.710461 11336 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36537 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:57:48 dist-test-slave-pxl6 krb5kdc[11105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.188@KRBTEST.COM
Sep 28 14:57:48 dist-test-slave-pxl6 krb5kdc[11105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607468, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.188@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:57:48.719246 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.711183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.20.116.190:36537)
0928 14:57:48.711251 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.711253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.711370 (+ 117us) server_negotiation.cc:407] Connection header received
0928 14:57:48.711406 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.711407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.711417 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.711484 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.711944 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.712395 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.713260 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.713457 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.713532 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.716288 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:48.716289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:48.716289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:48.716296 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:48.717881 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.718418 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.718419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.718419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.718468 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.718697 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.718698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.718698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.718868 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:48.718936 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:48.719116 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 14:57:48.719178 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:57:48.719290 11339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.708933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56110 (local address 127.20.116.188:37339)
0928 14:57:48.709143 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.709154 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.709188 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:57:48.709690 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.709699 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.709769 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.709925 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.710510 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.711832 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.712652 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.713064 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.713151 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.715598 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:48.715606 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:48.715606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:48.715614 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:48.717268 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.717667 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.717670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.717670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.717794 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.718111 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.718112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.718113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.718483 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:48.718656 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:48.718806 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 14:57:48.719031 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":83,"threads_started":1}
I20260928 14:57:48.719290 11340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.711128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.190:36537 (local address 127.0.0.1:53330)
0928 14:57:48.711267 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.711303 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:48.711368 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:48.711505 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:48.711507 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:48.711761 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:48.711833 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:48.711835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.712483 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.712484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:48.713157 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:48.713164 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.713556 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.713557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:48.713628 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.714127 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:48.714135 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:48.716187 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:48.717958 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:48.717963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:48.717965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:48.718346 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:48.718563 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:48.718564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:48.718564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:48.718629 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:48.718943 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:48.718948 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:48.718999 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 14:57:48.719086 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:57:48.719183 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":101,"threads_started":1}
I20260928 14:57:48.720243 11336 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:57:48 dist-test-slave-pxl6 krb5kdc[11105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607468, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.188@KRBTEST.COM for kudu/127.20.116.189@KRBTEST.COM
Sep 28 14:57:48 dist-test-slave-pxl6 krb5kdc[11105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.190@KRBTEST.COM
I20260928 14:57:48.727129 11340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.720504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.189:40935 (local address 127.0.0.1:57848)
0928 14:57:48.720537 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.720546 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:48.720576 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:48.720802 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:48.720803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:48.720962 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:48.721017 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:48.721019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.721663 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.721663 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:48.722149 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:48.722151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.722566 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.722567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:48.722642 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.723198 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:48.723202 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:48.724405 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:48.726113 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:48.726114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:48.726115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:48.726353 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:48.726573 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:48.726573 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:48.726574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:48.726639 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:48.726871 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:48.726871 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:48.726918 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 14:57:48.726988 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:57:48.727066 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 14:57:48.727233 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.720576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57848 (local address 127.20.116.189:40935)
0928 14:57:48.720649 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.720651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.720659 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:57:48.720689 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.720690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.720727 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.720786 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.721099 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.721584 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.722243 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.722468 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.722554 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.724493 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:48.724494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:48.724494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:48.724501 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:48.726024 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.726434 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.726435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.726435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.726486 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.726706 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.726707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.726707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.726808 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:48.726858 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:48.727030 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 14:57:48.727106 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:57:48.727746 11336 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:57:48.728113 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.720341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53342 (local address 127.20.116.190:36537)
0928 14:57:48.720382 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.720384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.720394 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:57:48.720415 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.720416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.720426 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.720483 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.720973 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.721482 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.722172 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.722398 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.722446 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.725273 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:48.725274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:48.725274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:48.725281 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:48.726879 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.727271 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.727272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.727272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.727331 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.727559 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.727560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.727560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.727682 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:48.727876 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:48.727920 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 14:57:48.728001 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 14:57:48 dist-test-slave-pxl6 krb5kdc[11105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607468, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.188@KRBTEST.COM for kudu/127.20.116.188@KRBTEST.COM
Sep 28 14:57:48 dist-test-slave-pxl6 krb5kdc[11105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.116.189@KRBTEST.COM
I20260928 14:57:48.735055 11340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.727998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:37339 (local address 127.0.0.1:56124)
0928 14:57:48.728047 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.728056 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:48.728088 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:48.728236 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:48.728237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:48.728390 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:48.728419 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:48.728420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.729336 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.729336 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:48.730048 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:48.730050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.730473 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.730474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:48.730512 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.730928 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:48.730931 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:48.732126 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:48.733843 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:48.733844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:48.733845 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:48.734072 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:48.734295 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:48.734297 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:48.734297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:48.734370 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:48.734690 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:48.734691 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:48.734730 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 14:57:48.734801 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 14:57:48.734872 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 14:57:48.735436 11339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.728060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56124 (local address 127.20.116.188:37339)
0928 14:57:48.728101 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.728103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.728110 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:57:48.728133 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.728134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.728144 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.728196 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.728513 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.729257 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.730127 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.730400 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.730469 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.732221 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:48.732222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:48.732223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:48.732230 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:48.733762 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.734142 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.734143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.734143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.734196 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.734448 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.734449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.734449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.734616 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:48.734691 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:48.734870 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 14:57:48.734944 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:57:48.736932 11336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce2fcbeb5e2b4457b0a6826010744455: Bootstrap starting.
I20260928 14:57:48.737114 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.728781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57864 (local address 127.20.116.189:40935)
0928 14:57:48.728821 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.728823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.728865 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:57:48.728931 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.728931 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.728938 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.728974 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.729513 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.730170 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.731097 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.731421 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.731511 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.734014 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:48.734015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:48.734015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:48.734021 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:48.735900 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.736286 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.736287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.736288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.736340 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.736543 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.736544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.736544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.736661 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:48.736841 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:48.736877 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 14:57:48.736949 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:57:48.738157 11336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce2fcbeb5e2b4457b0a6826010744455: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:48.738478 11336 log.cc:824] T 00000000000000000000000000000000 P ce2fcbeb5e2b4457b0a6826010744455: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:48.738646 20946 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:57:48.739267 11336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce2fcbeb5e2b4457b0a6826010744455: No bootstrap required, opened a new log
I20260928 14:57:48.739908 11160 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "88c5235003654094ae04e578c54cc32b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53342
I20260928 14:57:48.741636 11336 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce2fcbeb5e2b4457b0a6826010744455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88c5235003654094ae04e578c54cc32b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36537 } } peers { permanent_uuid: "6eb63e4ccd174039bbfe8572744db2ab" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } } peers { permanent_uuid: "ce2fcbeb5e2b4457b0a6826010744455" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } }
I20260928 14:57:48.741792 11336 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce2fcbeb5e2b4457b0a6826010744455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce2fcbeb5e2b4457b0a6826010744455, State: Initialized, Role: FOLLOWER
I20260928 14:57:48.741881 11336 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce2fcbeb5e2b4457b0a6826010744455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88c5235003654094ae04e578c54cc32b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36537 } } peers { permanent_uuid: "6eb63e4ccd174039bbfe8572744db2ab" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } } peers { permanent_uuid: "ce2fcbeb5e2b4457b0a6826010744455" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } }
I20260928 14:57:48.742157 11342 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce2fcbeb5e2b4457b0a6826010744455 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88c5235003654094ae04e578c54cc32b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36537 } } peers { permanent_uuid: "6eb63e4ccd174039bbfe8572744db2ab" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } } peers { permanent_uuid: "ce2fcbeb5e2b4457b0a6826010744455" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } } }
I20260928 14:57:48.742228 11336 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce2fcbeb5e2b4457b0a6826010744455 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:48.742290 11342 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce2fcbeb5e2b4457b0a6826010744455 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 14:57:48.744160 11353 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce2fcbeb5e2b4457b0a6826010744455: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:57:48.744244 11353 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:57:48.744406 11186 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } attempt: 2
I20260928 14:57:48.744540 11268 master_runner.cc:186] Error getting master registration for 127.20.116.190:36537: OK, instance_id { permanent_uuid: "88c5235003654094ae04e578c54cc32b" instance_seqno: 1790607468399044 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:57:48.746654 11268 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 14:57:48 dist-test-slave-pxl6 krb5kdc[11105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607468, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.190@KRBTEST.COM for kudu/127.20.116.188@KRBTEST.COM
I20260928 14:57:48.754397 11339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.746619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56140 (local address 127.20.116.188:37339)
0928 14:57:48.746733 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.746735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.746743 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:57:48.746762 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.746763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.746772 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.746827 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.747173 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.747882 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.748871 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.749159 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.749223 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.751507 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:48.751509 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:48.751510 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:48.751517 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:48.753141 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.753532 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.753534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.753534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.753598 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.753809 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.753810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.753810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.753995 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:48.754069 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:48.754228 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 14:57:48.754315 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 14:57:48.754388 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.744675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:37339 (local address 127.0.0.1:56140)
0928 14:57:48.744724 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.744736 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:48.744773 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:48.746842 (+ 2069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:48.746843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:48.747040 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:48.747091 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:48.747094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.747966 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.747968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:48.748786 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:48.748789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.749244 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.749245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:48.749307 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.749815 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:48.749821 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:48.751422 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:48.753238 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:48.753239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:48.753239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:48.753459 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:48.753670 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:48.753672 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:48.753672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:48.753743 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:48.754065 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:48.754066 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:48.754112 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 14:57:48.754181 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 14:57:48.754278 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 14:57:48.756568 11186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b: Bootstrap starting.
I20260928 14:57:48.757357 11186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:48.757699 11186 log.cc:824] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:48.758646 11186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b: No bootstrap required, opened a new log
I20260928 14:57:48.760010 11186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88c5235003654094ae04e578c54cc32b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36537 } } peers { permanent_uuid: "6eb63e4ccd174039bbfe8572744db2ab" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } } peers { permanent_uuid: "ce2fcbeb5e2b4457b0a6826010744455" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } }
I20260928 14:57:48.760159 11186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88c5235003654094ae04e578c54cc32b, State: Initialized, Role: FOLLOWER
I20260928 14:57:48.760249 11186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88c5235003654094ae04e578c54cc32b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36537 } } peers { permanent_uuid: "6eb63e4ccd174039bbfe8572744db2ab" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } } peers { permanent_uuid: "ce2fcbeb5e2b4457b0a6826010744455" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } }
I20260928 14:57:48.760561 11186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:48.760515 11356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88c5235003654094ae04e578c54cc32b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36537 } } peers { permanent_uuid: "6eb63e4ccd174039bbfe8572744db2ab" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } } peers { permanent_uuid: "ce2fcbeb5e2b4457b0a6826010744455" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } } }
I20260928 14:57:48.760586 11160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 14:57:48.760659 11356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:57:48.760715 11160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:48.761504 11160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88c5235003654094ae04e578c54cc32b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36537 } } peers { permanent_uuid: "6eb63e4ccd174039bbfe8572744db2ab" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } } peers { permanent_uuid: "ce2fcbeb5e2b4457b0a6826010744455" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } }
I20260928 14:57:48.762015 11310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88c5235003654094ae04e578c54cc32b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ce2fcbeb5e2b4457b0a6826010744455"
I20260928 14:57:48.762168 11310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce2fcbeb5e2b4457b0a6826010744455 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:48.762574 11160 leader_election.cc:288] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b [CANDIDATE]: Term 1 election: Requested vote from peers 6eb63e4ccd174039bbfe8572744db2ab (127.20.116.189:40935), ce2fcbeb5e2b4457b0a6826010744455 (127.20.116.188:37339)
I20260928 14:57:48.763151 11310 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ce2fcbeb5e2b4457b0a6826010744455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88c5235003654094ae04e578c54cc32b in term 1.
I20260928 14:57:48.763399 11136 leader_election.cc:302] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88c5235003654094ae04e578c54cc32b, ce2fcbeb5e2b4457b0a6826010744455; no voters:
I20260928 14:57:48.763527 11356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:48.763682 11356 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b [term 1 LEADER]: Becoming Leader. State: Replica: 88c5235003654094ae04e578c54cc32b, State: Running, Role: LEADER
I20260928 14:57:48.763770 11356 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88c5235003654094ae04e578c54cc32b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36537 } } peers { permanent_uuid: "6eb63e4ccd174039bbfe8572744db2ab" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } } peers { permanent_uuid: "ce2fcbeb5e2b4457b0a6826010744455" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } }
I20260928 14:57:48.764197 11356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88c5235003654094ae04e578c54cc32b. Latest consensus state: current_term: 1 leader_uuid: "88c5235003654094ae04e578c54cc32b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88c5235003654094ae04e578c54cc32b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36537 } } peers { permanent_uuid: "6eb63e4ccd174039bbfe8572744db2ab" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } } peers { permanent_uuid: "ce2fcbeb5e2b4457b0a6826010744455" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } } }
I20260928 14:57:48.764295 11356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:48.764503 11374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:48.765044 11374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:48.763230 11118 master_runner.cc:186] Error getting master registration for 127.20.116.189:40935: OK, instance_id { permanent_uuid: "6eb63e4ccd174039bbfe8572744db2ab" instance_seqno: 1790607468549145 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:57:48.765898 11118 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:57:48.771458 11310 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ce2fcbeb5e2b4457b0a6826010744455 [term 1 FOLLOWER]: Refusing update from remote peer 88c5235003654094ae04e578c54cc32b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:48.771833 11372 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce2fcbeb5e2b4457b0a6826010744455" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:48.774153 11342 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce2fcbeb5e2b4457b0a6826010744455 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88c5235003654094ae04e578c54cc32b. Latest consensus state: current_term: 1 leader_uuid: "88c5235003654094ae04e578c54cc32b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88c5235003654094ae04e578c54cc32b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36537 } } peers { permanent_uuid: "6eb63e4ccd174039bbfe8572744db2ab" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } } peers { permanent_uuid: "ce2fcbeb5e2b4457b0a6826010744455" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } } }
I20260928 14:57:48.774259 11342 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce2fcbeb5e2b4457b0a6826010744455 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:57:48.775789 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.766821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53356 (local address 127.20.116.190:36537)
0928 14:57:48.766861 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.766863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.767698 (+ 835us) server_negotiation.cc:407] Connection header received
0928 14:57:48.767803 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.767804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.767815 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.767847 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.768689 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.769478 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.770577 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.770830 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.770874 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.772803 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:48.772804 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:48.772804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:48.772812 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:48.774506 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.774927 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.774928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.774929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.774983 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.775185 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.775186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.775186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.775327 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:48.775401 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:48.775615 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 14:57:48.775697 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:57:48.784091 11261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } attempt: 3
I20260928 14:57:48.784237 11372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88c5235003654094ae04e578c54cc32b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88c5235003654094ae04e578c54cc32b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36537 } } peers { permanent_uuid: "6eb63e4ccd174039bbfe8572744db2ab" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } } peers { permanent_uuid: "ce2fcbeb5e2b4457b0a6826010744455" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } } }
I20260928 14:57:48.784365 11372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:48.784453 11376 mvcc.cc:205] Tried to move back new op lower bound from 7334328192085286912 to 7334328192057200640. Current Snapshot: MvccSnapshot[applied={T|T < 7334328192085286912}]
I20260928 14:57:48.785283 11342 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce2fcbeb5e2b4457b0a6826010744455 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88c5235003654094ae04e578c54cc32b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88c5235003654094ae04e578c54cc32b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36537 } } peers { permanent_uuid: "6eb63e4ccd174039bbfe8572744db2ab" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } } peers { permanent_uuid: "ce2fcbeb5e2b4457b0a6826010744455" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } } }
I20260928 14:57:48.785389 11342 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce2fcbeb5e2b4457b0a6826010744455 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:57:48.785758 11380 mvcc.cc:205] Tried to move back new op lower bound from 7334328192085286912 to 7334328192057200640. Current Snapshot: MvccSnapshot[applied={T|T < 7334328192085286912}]
I20260928 14:57:48.787539 11339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.767050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56148 (local address 127.20.116.188:37339)
0928 14:57:48.767083 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.767085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.776026 (+ 8941us) server_negotiation.cc:407] Connection header received
0928 14:57:48.776132 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.776134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.776146 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.776180 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.777272 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.777995 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.779060 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.779374 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.779416 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.781930 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:48.781932 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:48.781932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:48.781940 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:48.783953 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.785174 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.785187 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.785188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.785251 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.786765 (+ 1514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.786767 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.786767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.786902 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:48.786989 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:48.787367 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 14:57:48.787447 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 14:57:48.788105 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.767406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57870 (local address 127.20.116.189:40935)
0928 14:57:48.767466 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.767468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.776319 (+ 8851us) server_negotiation.cc:407] Connection header received
0928 14:57:48.776423 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.776425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.776439 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.776471 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.779795 (+ 3324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.780532 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.781612 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.782394 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.782440 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.783824 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:48.783825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:48.783826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:48.783834 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:48.786853 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.787218 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.787220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.787220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.787275 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.787461 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.787461 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.787462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.787612 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:48.787695 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:48.787936 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 14:57:48.788015 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:57:48.790879 11374 catalog_manager.cc:1347] Generated new cluster ID: c4d244f71a504285925e137f231d9714
I20260928 14:57:48.790937 11374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:57:48 dist-test-slave-pxl6 krb5kdc[11105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607468, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.116.189@KRBTEST.COM for kudu/127.20.116.188@KRBTEST.COM
I20260928 14:57:48.797194 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.784347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.116.188:37339 (local address 127.0.0.1:56152)
0928 14:57:48.784436 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:48.784446 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:48.784482 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:48.786437 (+ 1955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:48.786439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:48.786633 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:48.786676 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:48.786678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.789770 (+ 3092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.789772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:48.790621 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:48.790623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.791164 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.791165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:48.791235 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.791618 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:48.791623 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:48.793052 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:48.795807 (+ 2755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:48.795808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:48.795809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:48.796040 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:48.796349 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:48.796350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:48.796350 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:48.796420 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:48.796882 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:48.796883 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:48.796932 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 14:57:48.796980 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 14:57:48.797087 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 14:57:48.797782 11383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:48.784548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56152 (local address 127.20.116.188:37339)
0928 14:57:48.786294 (+ 1746us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.786296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.786304 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:57:48.786324 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.786324 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.786334 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.788772 (+ 2438us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.788806 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.789641 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.790730 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:48.791080 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:48.791377 (+ 297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:48.793147 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:48.793148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:48.793149 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:48.793156 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:48.795690 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.796137 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.796139 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.796139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.796256 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:48.796504 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:48.796505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:48.796505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:48.796767 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:48.797400 (+ 633us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:48.797487 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 14:57:48.797620 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1726,"thread_start_us":76,"threads_started":1}
I20260928 14:57:48.799490 11261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb63e4ccd174039bbfe8572744db2ab: Bootstrap starting.
I20260928 14:57:48.800308 11261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6eb63e4ccd174039bbfe8572744db2ab: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:48.800611 11261 log.cc:824] T 00000000000000000000000000000000 P 6eb63e4ccd174039bbfe8572744db2ab: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:48.801481 11261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb63e4ccd174039bbfe8572744db2ab: No bootstrap required, opened a new log
I20260928 14:57:48.803635 11261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6eb63e4ccd174039bbfe8572744db2ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88c5235003654094ae04e578c54cc32b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36537 } } peers { permanent_uuid: "6eb63e4ccd174039bbfe8572744db2ab" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } } peers { permanent_uuid: "ce2fcbeb5e2b4457b0a6826010744455" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } }
I20260928 14:57:48.803833 11261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6eb63e4ccd174039bbfe8572744db2ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eb63e4ccd174039bbfe8572744db2ab, State: Initialized, Role: FOLLOWER
I20260928 14:57:48.803992 11261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6eb63e4ccd174039bbfe8572744db2ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88c5235003654094ae04e578c54cc32b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36537 } } peers { permanent_uuid: "6eb63e4ccd174039bbfe8572744db2ab" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } } peers { permanent_uuid: "ce2fcbeb5e2b4457b0a6826010744455" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } }
I20260928 14:57:48.804433 11261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6eb63e4ccd174039bbfe8572744db2ab [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:48.804476 11235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88c5235003654094ae04e578c54cc32b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6eb63e4ccd174039bbfe8572744db2ab"
I20260928 14:57:48.804570 11235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6eb63e4ccd174039bbfe8572744db2ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:48.805475 11235 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6eb63e4ccd174039bbfe8572744db2ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88c5235003654094ae04e578c54cc32b in term 1.
I20260928 14:57:48.805697 11234 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6eb63e4ccd174039bbfe8572744db2ab [term 1 FOLLOWER]: Refusing update from remote peer 88c5235003654094ae04e578c54cc32b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:57:48.805999 11356 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6eb63e4ccd174039bbfe8572744db2ab" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:57:48.804399 11386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb63e4ccd174039bbfe8572744db2ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88c5235003654094ae04e578c54cc32b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36537 } } peers { permanent_uuid: "6eb63e4ccd174039bbfe8572744db2ab" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } } peers { permanent_uuid: "ce2fcbeb5e2b4457b0a6826010744455" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } } }
I20260928 14:57:48.806249 11386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb63e4ccd174039bbfe8572744db2ab [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:57:48.806573 11374 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:48.807385 11374 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:48.808307 11386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb63e4ccd174039bbfe8572744db2ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88c5235003654094ae04e578c54cc32b. Latest consensus state: current_term: 1 leader_uuid: "88c5235003654094ae04e578c54cc32b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88c5235003654094ae04e578c54cc32b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36537 } } peers { permanent_uuid: "6eb63e4ccd174039bbfe8572744db2ab" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } } peers { permanent_uuid: "ce2fcbeb5e2b4457b0a6826010744455" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } } }
I20260928 14:57:48.808633 11386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb63e4ccd174039bbfe8572744db2ab [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:57:48.808683 11386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb63e4ccd174039bbfe8572744db2ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88c5235003654094ae04e578c54cc32b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88c5235003654094ae04e578c54cc32b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36537 } } peers { permanent_uuid: "6eb63e4ccd174039bbfe8572744db2ab" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } } peers { permanent_uuid: "ce2fcbeb5e2b4457b0a6826010744455" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } } }
I20260928 14:57:48.808890 11386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb63e4ccd174039bbfe8572744db2ab [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:57:48.811368 11397 mvcc.cc:205] Tried to move back new op lower bound from 7334328192206422016 to 7334328192057200640. Current Snapshot: MvccSnapshot[applied={T|T < 7334328192085286912}]
I20260928 14:57:48.815837 11400 catalog_manager.cc:1259] Loaded cluster ID: c4d244f71a504285925e137f231d9714
I20260928 14:57:48.815896 11400 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6eb63e4ccd174039bbfe8572744db2ab: loading cluster ID for follower catalog manager: success
I20260928 14:57:48.817163 11400 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6eb63e4ccd174039bbfe8572744db2ab: acquiring CA information for follower catalog manager: success
W20260928 14:57:48.817346 11400 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260928 14:57:48.821257 11356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88c5235003654094ae04e578c54cc32b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88c5235003654094ae04e578c54cc32b" member_type: VOTER last_known_addr { host: "127.20.116.190" port: 36537 } } peers { permanent_uuid: "6eb63e4ccd174039bbfe8572744db2ab" member_type: VOTER last_known_addr { host: "127.20.116.189" port: 40935 } } peers { permanent_uuid: "ce2fcbeb5e2b4457b0a6826010744455" member_type: VOTER last_known_addr { host: "127.20.116.188" port: 37339 } } }
I20260928 14:57:48.821377 11356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:48.822376 11374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88c5235003654094ae04e578c54cc32b: Generated new TSK 0
I20260928 14:57:48.822621 11374 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:57:48.849519 11265 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:48.848079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57876 (local address 127.20.116.189:40935)
0928 14:57:48.848127 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.848129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.848832 (+ 703us) server_negotiation.cc:407] Connection header received
0928 14:57:48.848943 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.848944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.848956 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.848987 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.849444 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57876: BlockingRecv error: recv got EOF from 127.0.0.1:57876 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 14:57:48.850296 11383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:48.848479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56168 (local address 127.20.116.188:37339)
0928 14:57:48.848520 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.848522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.849781 (+ 1259us) server_negotiation.cc:407] Connection header received
0928 14:57:48.849879 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.849881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.849893 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.849923 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.850234 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56168: BlockingRecv error: recv got EOF from 127.0.0.1:56168 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 14:57:48.851450 11189 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:48.850712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53358 (local address 127.20.116.190:36537)
0928 14:57:48.850753 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.850756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.850886 (+ 130us) server_negotiation.cc:407] Connection header received
0928 14:57:48.850983 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.850984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.850995 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.851025 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.851385 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53358: BlockingRecv error: recv got EOF from 127.0.0.1:53358 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 14:57:48.853853 11265 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:48.853009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57880 (local address 127.20.116.189:40935)
0928 14:57:48.853044 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.853046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.853300 (+ 254us) server_negotiation.cc:407] Connection header received
0928 14:57:48.853415 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.853416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.853436 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.853465 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.853801 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57880: BlockingRecv error: recv got EOF from 127.0.0.1:57880 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 14:57:48.856825 11265 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:48.856044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57892 (local address 127.20.116.189:40935)
0928 14:57:48.856078 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.856080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.856314 (+ 234us) server_negotiation.cc:407] Connection header received
0928 14:57:48.856410 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.856411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.856421 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.856449 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.856779 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57892: BlockingRecv error: recv got EOF from 127.0.0.1:57892 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 14:57:48.860698 11265 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:48.859931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57898 (local address 127.20.116.189:40935)
0928 14:57:48.859966 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.859967 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.860197 (+ 230us) server_negotiation.cc:407] Connection header received
0928 14:57:48.860294 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.860295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.860307 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.860337 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.860652 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57898: BlockingRecv error: recv got EOF from 127.0.0.1:57898 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 14:57:48.866565 11265 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:48.865736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57910 (local address 127.20.116.189:40935)
0928 14:57:48.865771 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.865773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.866007 (+ 234us) server_negotiation.cc:407] Connection header received
0928 14:57:48.866117 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.866118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.866128 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.866157 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.866482 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57910: BlockingRecv error: recv got EOF from 127.0.0.1:57910 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 14:57:48.876565 11265 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:48.875709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57924 (local address 127.20.116.189:40935)
0928 14:57:48.875751 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.875753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.876018 (+ 265us) server_negotiation.cc:407] Connection header received
0928 14:57:48.876125 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.876126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.876137 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.876166 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.876507 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57924: BlockingRecv error: recv got EOF from 127.0.0.1:57924 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 14:57:48.895118 11265 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:48.894053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57938 (local address 127.20.116.189:40935)
0928 14:57:48.894099 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.894101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.894428 (+ 327us) server_negotiation.cc:407] Connection header received
0928 14:57:48.894593 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.894595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.894608 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.894651 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.895054 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57938: BlockingRecv error: recv got EOF from 127.0.0.1:57938 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 14:57:48.929540 11265 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:48.928528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57942 (local address 127.20.116.189:40935)
0928 14:57:48.928577 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.928579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.928904 (+ 325us) server_negotiation.cc:407] Connection header received
0928 14:57:48.929025 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.929027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.929040 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.929071 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.929477 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57942: BlockingRecv error: recv got EOF from 127.0.0.1:57942 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 14:57:48.995940 11265 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:48.995063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57948 (local address 127.20.116.189:40935)
0928 14:57:48.995174 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:48.995177 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:48.995285 (+ 108us) server_negotiation.cc:407] Connection header received
0928 14:57:48.995390 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:48.995391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:48.995405 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:48.995485 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:48.995874 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57948: BlockingRecv error: recv got EOF from 127.0.0.1:57948 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 14:57:49.126199 11265 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:49.125314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57954 (local address 127.20.116.189:40935)
0928 14:57:49.125413 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:49.125415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:49.125513 (+ 98us) server_negotiation.cc:407] Connection header received
0928 14:57:49.125620 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:49.125621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:49.125633 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:49.125719 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:49.126125 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57954: BlockingRecv error: recv got EOF from 127.0.0.1:57954 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 14:57:49.384285 11265 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:57:49.383494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57970 (local address 127.20.116.189:40935)
0928 14:57:49.383571 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:49.383573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:49.383604 (+ 31us) server_negotiation.cc:407] Connection header received
0928 14:57:49.383682 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:49.383683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:49.383695 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:49.383755 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:49.384175 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57970: BlockingRecv error: recv got EOF from 127.0.0.1:57970 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:57:49.744706 11353 catalog_manager.cc:1259] Loaded cluster ID: c4d244f71a504285925e137f231d9714
I20260928 14:57:49.744786 11353 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ce2fcbeb5e2b4457b0a6826010744455: loading cluster ID for follower catalog manager: success
I20260928 14:57:49.745700 11353 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ce2fcbeb5e2b4457b0a6826010744455: acquiring CA information for follower catalog manager: success
I20260928 14:57:49.745972 11353 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ce2fcbeb5e2b4457b0a6826010744455: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:57:49.817978 11400 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6eb63e4ccd174039bbfe8572744db2ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:57:49.900310 11468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:49.897432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57972 (local address 127.20.116.189:40935)
0928 14:57:49.897541 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:49.897543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:49.897556 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:57:49.897691 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:49.897692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:49.897776 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:49.897872 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:49.898029 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:49.898554 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:49.899295 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:49.899496 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:49.899549 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:49.899814 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:49.899921 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:49.899981 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:49.900169 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 14:57:49.900187 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":61,"threads_started":1}
I20260928 14:57:49.904549 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:49.902120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56184 (local address 127.20.116.188:37339)
0928 14:57:49.902226 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:49.902228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:49.902237 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:57:49.902285 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:49.902286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:49.902323 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:49.902393 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:49.902493 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:49.903006 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:49.903743 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:49.904023 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:49.904073 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:49.904231 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:49.904326 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:49.904385 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:49.904469 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 14:57:49.904487 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":63,"threads_started":1}
I20260928 14:57:49.909185 11468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:49.906175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57986 (local address 127.20.116.189:40935)
0928 14:57:49.906335 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:49.906337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:49.906577 (+ 240us) server_negotiation.cc:407] Connection header received
0928 14:57:49.906687 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:49.906688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:49.906701 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:49.906730 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:49.906926 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:49.907632 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:49.908385 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:49.908675 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:49.908746 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:49.908868 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:49.908902 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:49.908952 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:49.909054 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 14:57:49.909075 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260928 14:57:49.909325 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:49.906341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56194 (local address 127.20.116.188:37339)
0928 14:57:49.906498 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:49.906501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:49.906629 (+ 128us) server_negotiation.cc:407] Connection header received
0928 14:57:49.906731 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:49.906731 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:49.906742 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:49.906770 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:49.906961 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:49.907407 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:49.908461 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:49.908751 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:49.908805 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:49.908956 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:49.908996 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:49.909043 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:49.909233 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 14:57:49.909256 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260928 14:57:49.909871 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:49.906246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53360 (local address 127.20.116.190:36537)
0928 14:57:49.906353 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:49.906355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:49.906365 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:57:49.906455 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:49.906456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:49.906513 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:49.906626 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:57:49.906798 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:49.907592 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:49.908620 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:49.908959 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:49.909039 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:49.909183 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:57:49.909311 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:57:49.909374 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:49.909557 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 14:57:49.909765 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":53,"threads_started":1}
I20260928 14:57:49.916918 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 11118
I20260928 14:57:49.922821 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 11193
I20260928 14:57:49.927867 20946 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOCNdC/build/release/bin/kudu with pid 11268
2026-09-28T14:57:49Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2820 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5624 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (277139 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST